python - Is POS tagging deterministic? -


i have been trying wrap head around why happening hoping can shed light on this. trying tag following text:

ae0.475      x  mod  ae0.842      x  mod ae0.842      x  mod  ae0.775      x  mod

using following code:

import nltk  file = open("test", "r")  line in file:         words = line.strip().split(' ')         words = [word.strip() word in words if word != '']         tags = nltk.pos_tag(words)         pos = [tags[x][1] x in range(len(tags))]         key = ' '.join(pos)         print words, " : ", key

and getting following result:

['ae0.475', 'x', 'mod']  :  nn nnp nn ['ae0.842', 'x', 'mod']  :  -none- nnp nn ['ae0.842', 'x', 'mod']  :  -none- nnp nn ['ae0.775', 'x', 'mod']  :  nn nnp nn

and don't it. know reason inconsistency? not particular accuracy pos tagging because attempting extract templates seems using different tags @ different instances word looks "almost" same.

as solution, replaced numbers 1 , solved problem:

['ae1.111', 'x', 'mod']  :  nn nnp nn ['ae1.111', 'x', 'mod']  :  nn nnp nn ['ae1.111', 'x', 'mod']  :  nn nnp nn ['ae1.111', 'x', 'mod']  :  nn nnp nn

but curious why tagged instance different tags in first case. suggestions?

my best effort understand uncovered this not using whole brown corpus:

note words tagger has not seen before, such decried, receive tag of none.

so, guess looks ae1.111 must appear in corpus file, nothing ae0.842. that's kind of weird, that's reasoning giving -none- tag.

edit: got super-curious, downloaded brown corpus myself, , plain-text-searched inside it. number 111 appears in 34 times, , number 842 appears 4 times. 842 appears either in middle of dollar amounts or last 3 digits of year, , 111 appears many times on own page number. 775 appears once page number.

so, i'm going make conjecture, because of benford's law, end matching numbers start 1s, 2s, , 3s more numbers start 8s or 9s, since these more page numbers of random page cited in book. i'd interested in finding out if that's true (but not interested enough myself, of course!).


Comments

Post a Comment

Popular posts from this blog

c++ - Is it possible to compile a VST on linux? -

for class project i'm attempting write vst plugin backed cuda. current cuda workflow on linux box, i'd prefer compile , link there. according wikipedia , should possible (i couldn't find steinberg documentation relevant linux) can't find makefile or instructions on how build if aren't using xcode or visual studio. i'm vst 3 sdk doesn't support linux. when try compile plugin under linux, error: ./base/source/fatomic.cpp:39:30: fatal error: libkern/osatomic.h: no such file or directory this issue caused following code in "vst3 sdk/base/source/fatomic.cpp" #if mac #include <libkern/osatomic.h> #if mac_os_x_version_min_required > mac_os_x_version_10_4 #define native_atomic_type (volatile int32_t*) #else #define native_atomic_type (int32_t*) #endif #elif windows #include <windows.h> #endif but hope compiling under linux work vst sdk 2.4. reading. jvstwrapper seems run on lin...
Read more

c# - SharpSVN - How to get the previous revision? -

i trying find efficient way previous revision of file text comparison using sharpsvn. using (svnclient c = new svnclient()) { c.authentication.defaultcredentials = new networkcredential( configurationmanager.appsettings.get("svnserviceusername") , configurationmanager.appsettings.get("svnservicepassword") , configurationmanager.appsettings.get("svnservicedomain") ); c.authentication.sslservertrusthandlers += new eventhandler<svnsslservertrusteventargs>(authentication_sslservertrusthandlers); collection<svnfileversioneventargs> fileversioncollection = new collection<svnfileversioneventargs>(); svnrevisionrange range = new svnrevisionrange(0, this.hooks.revision); svnfileversionsargs args = new svnfileversionsargs(); args.retrieveproperties = true; args.range = range; foreach (svnchangeitem item in log.changedpaths) { string path = this.repositorypath + it...
Read more

php cli reading files and how to fix it? -

while using php cli in ubuntu got apache web server cant read file can while using in browser. , error shows nothing , next used code testing purposes : <? $handle = fopen("testfile.txt", "r"); if ($handle) { while (($buffer = fgets($handle, 4096)) !== false) { echo $buffer; } if (!feof($handle)) { echo "error: unexpected fgets() fail\n"; } if(!is_readable($handle) { die("not readable"); } fclose($handle); } ?> how fix ? edit : after removing '@' before fopen got following error fopen(testfile.txt): failed open stream: no such file or directory in /var/www/log/ka.php on line 2 when run script through cli, uses shell's working directory, in opposition file's directory (which apache hands current directory). important you, since fopen call depends on relative path; , relative paths resolved relatively working directory, not script. to have script behave apache, eit...
Read more