Tokenizer

Collaboration diagram for Tokenizer:

Detects linguistic tokens.

Access Adapter Interface: de.uni_koeln.spinfo.tesla.roles.tokenizer.access.ITokenAccessAdapter

Data Object Interface: de.uni_koeln.spinfo.tesla.roles.tokenizer.data.IToken

Produced by:

• Role 'Tokens' of component 'CHILDES Reader' (de.uni_koeln.spinfo.tesla.component.reader.ChildesReader)
• Role 'Tokens' of component 'BNC Reader' (de.uni_koeln.spinfo.tesla.component.reader.BNCReader)
• Role 'Tokens' of component 'TüBa Corpus Reader' (de.uni_koeln.spinfo.tesla.component.reader.TueBaXmlReader)
• Role 'Dummy' of component 'Sequence Metrics' (org.schwiebert.tesla.rangecomparator.SequenceMetrics)
• Role 'Dummy' of component 'Group Similarity Evaluator' (de.uni_koeln.spinfo.tesla.component.groupeval.PurityValidatorComponent)
• Role 'HibernateOut' of component 'Mean Component' (de.uni_koeln.spinfo.tesla.component.meancomponent.MeanComponent)
• Role 'TunguskaOut' of component 'Mean Component' (de.uni_koeln.spinfo.tesla.component.meancomponent.MeanComponent)
• Role 'Substitutions' of component 'Cryptographic Substitution' (de.uni_koeln.spinfo.cryptology.teslacomponent.CryptographicSubstitutionComponent)
• Role 'Tokens' of component 'Simple Tokenizer' (de.uni_koeln.spinfo.tesla.component.simpletokenizer.SimpleTokenizer)
• Role 'Tokens' of component 'Token rewriter (Old)' (de.uni_koeln.spinfo.tesla.component.misc.TokenRewriter)
• Role 'Tokens' of component 'SPre' (de.uni_koeln.spinfo.tesla.component.spre.SPre2Component)
• Role 'Words' of component 'Tiger Corpus Reader' (de.uni_koeln.spinfo.tesla.component.reader.TigerCorpusReader)
• Role 'Tokens' of component 'Jython RegEx Demo' (de.uni_koeln.spinfo.tesla.component.jython.regex.JythonRegexComponent)
• Role 'Morphemes' of component 'Morphemizer' (de.uni_koeln.spinfo.formanalysis.teslacomponents.SimpleMorphemizerComponent)
• Role 'Tokens' of component 'Token rewriter' (de.uni_koeln.spinfo.tesla.component.sequencer.TokenRewriter)
• Role 'Tokens' of component 'Brown TEI Reader' (de.uni_koeln.spinfo.tesla.component.reader.BrownTEIReader)

Can be consumed by:

• Role 'Sequences'( Anchored Element Generator) of component 'SOG (Analyze)' (de.uni_koeln.spinfo.tesla.component.misc.FindProperties)
• Role 'Sequence Items'( Anchored Element Generator) of component 'SOG (Analyze)' (de.uni_koeln.spinfo.tesla.component.misc.FindProperties)
• Role 'Sequence Items'( Anchored Element Generator) of component 'Random Align' (de.uni_koeln.spinfo.tesla.component.abl.RandomLeftRightComponent)
• Role 'Sequences'( Anchored Element Generator) of component 'Random Align' (de.uni_koeln.spinfo.tesla.component.abl.RandomLeftRightComponent)
• Role 'Symbols'( Anchored Element Generator) of component 'Snowball Stemmer' (de.uni_koeln.spinfo.tesla.component.snowball.TeslaSnowballStemmer)
• Role 'New Items'( Anchored Element Generator) of component 'Simple Type Frequencies' (de.uni_koeln.spinfo.tesla.component.sequencer.TypeFrequencies)
• Role 'Sequences'( Anchored Element Generator) of component 'Hypothesis Generator' (de.uni_koeln.spinfo.tesla.component.paradigms.HypothesisEvaluator)
• Role 'Sequence Items'( Anchored Element Generator) of component 'Hypothesis Generator' (de.uni_koeln.spinfo.tesla.component.paradigms.HypothesisEvaluator)
• Role 'Words'( Tokenizer) of component 'Jython Porter Stemmer' (de.uni_koeln.spinfo.tesla.component.jython.stemmer.JythonPorterStemmer)
• Role 'Words'( Tokenizer) of component 'Graphemizer' (de.uni_koeln.spinfo.formanalysis.teslacomponents.GraphemizerComponent)
• Role 'Filter'( Anchored Element Generator) of component 'Generic Range Filter' (de.uni_koeln.spinfo.tesla.component.filter.GenericRangeFilter)
• Role 'Words'( Tokenizer) of component 'Corpus Statistics' (de.uni_koeln.spinfo.tesla.component.corpusstatistics.CorpusStatisticsComponent)
• Role 'Tokens'( Tokenizer) of component 'Stanford Parser' (de.uni_koeln.spinfo.tesla.component.stanfordparser.StanfordParser)
• Role 'Sequences'( Anchored Element Generator) of component 'Sequence Metrics' (org.schwiebert.tesla.rangecomparator.SequenceMetrics)
• Role 'To Count'( Anchored Element Generator) of component 'Sequence Metrics' (org.schwiebert.tesla.rangecomparator.SequenceMetrics)
• Role 'Sequence Items'( Anchored Element Generator) of component 'Sequence Metrics' (org.schwiebert.tesla.rangecomparator.SequenceMetrics)
• Role 'Reference'( Anchored Element Generator) of component 'Group Similarity Evaluator' (de.uni_koeln.spinfo.tesla.component.groupeval.PurityValidatorComponent)
• Role 'Words'( Tokenizer) of component 'Repeated Words Detector' (de.uni_koeln.spinfo.tesla.component.corpusstatistics.RepeatedWordsDetector)
• Role 'Data'( Annotator) of component 'Mean Component' (de.uni_koeln.spinfo.tesla.component.meancomponent.MeanComponent)
• Role 'Words'( Tokenizer) of component 'Tree Tagger Wrapper (non-commercial usage only)' (de.uni_koeln.spinfo.tesla.component.sttstagger.SttsTagger)
• Role 'Study Corpus Words'( Tokenizer) of component 'Corpus Log Likelihood' (de.uni_koeln.spinfo.tesla.component.statistics.CorpusLogLikelihoodCalculator)
• Role 'Elements'( Anchored Element Generator) of component 'Simple Collocation Detector' (de.uni_koeln.spinfo.tesla.component.collocations.CollocationDetector)
• Role 'Study Corpus Words'( Tokenizer) of component 'Letter Serial Correlation (LSC)' (de.spinfo.tesla.component.LSC.LSCCalculatorComponent)
• Role 'Sequences'( Anchored Element Generator) of component 'Boundaries Detector' (de.uni_koeln.spinfo.tesla.component.alignhypothesis.HypothesisGenerator)
• Role 'Sequence Items'( Anchored Element Generator) of component 'Boundaries Detector' (de.uni_koeln.spinfo.tesla.component.alignhypothesis.HypothesisGenerator)
• Role 'Words'( Tokenizer) of component 'Cryptographic Substitution' (de.uni_koeln.spinfo.cryptology.teslacomponent.CryptographicSubstitutionComponent)
• Role 'Corpus Words'( Tokenizer) of component 'TF IDF Table' (de.uni_koeln.spinfo.tesla.component.statistics.TfIdfTable)
• Role 'Filter'( Anchored Element Generator) of component 'Range Filter' (de.uni_koeln.spinfo.tesla.component.filter.RangeFilter)
• Role 'Reference'( Anchored Element Generator) of component 'Range Filter' (de.uni_koeln.spinfo.tesla.component.filter.RangeFilter)
• Role 'Items'( Sequence Annotator) of component 'Word Vector Generator' (de.uni_koeln.spinfo.tesla.component.vectorgenerator.FilteringDoubleVectorGenerator)
• Role 'Sequence Items'( Anchored Element Generator) of component 'ABL Align' (de.uni_koeln.spinfo.tesla.component.abl.AblAlignComponent)
• Role 'Sequences'( Anchored Element Generator) of component 'ABL Align' (de.uni_koeln.spinfo.tesla.component.abl.AblAlignComponent)
• Role 'Tokens'( Tokenizer) of component 'Tree Label Converter' (de.uni_koeln.spinfo.tesla.component.stanfordparser.converter.TreeLabelConverter)
• Role 'Words'( Tokenizer) of component 'Token rewriter (Old)' (de.uni_koeln.spinfo.tesla.component.misc.TokenRewriter)
• Role 'Sequences'( Anchored Element Generator) of component 'Random Align' (de.uni_koeln.spinfo.tesla.component.abl.RandomAlignComponent)
• Role 'Constituents'( Anchored Element Generator) of component 'Random Align' (de.uni_koeln.spinfo.tesla.component.abl.RandomAlignComponent)
• Role 'Sequence Items'( Anchored Element Generator) of component 'Random Align' (de.uni_koeln.spinfo.tesla.component.abl.RandomAlignComponent)
• Role 'Sequences'( Anchored Element Generator) of component 'N-Gram Tree Generator' (de.uni_koeln.spinfo.tesla.component.ngramtree.NGramTreeGenerator)
• Role 'Symbols'( Anchored Element Generator) of component 'N-Gram Tree Generator' (de.uni_koeln.spinfo.tesla.component.ngramtree.NGramTreeGenerator)
• Role 'Sequences'( Anchored Element Generator) of component 'SOG (Filter)' (de.uni_koeln.spinfo.tesla.component.misc.TestFilter)
• Role 'Sequence Items'( Anchored Element Generator) of component 'SOG (Filter)' (de.uni_koeln.spinfo.tesla.component.misc.TestFilter)
• Role 'Words'( Tokenizer) of component 'Jython Regex Stemmer' (de.uni_koeln.spinfo.tesla.component.jython.stemmer.JythonRegexStemmer)
• Role 'Base Sequence'( Anchored Element Generator) of component 'Order Based Sequencer' (de.uni_koeln.spinfo.tesla.component.sequencer.OrderBasedSequencer)
• Role 'Replacements'( Anchored Element Generator) of component 'Order Based Sequencer' (de.uni_koeln.spinfo.tesla.component.sequencer.OrderBasedSequencer)
• Role 'Sequences'( Anchored Element Generator) of component 'ABL Cluster' (de.uni_koeln.spinfo.tesla.component.abl.AblClusterComponent)
• Role 'Sequence Items'( Anchored Element Generator) of component 'ABL Cluster' (de.uni_koeln.spinfo.tesla.component.abl.AblClusterComponent)
• Role 'Words'( Tokenizer) of component 'Geographic Location Detector' (de.uni_koeln.spinfo.tesla.component.geolocations.GeoNamesDetector)
• Role 'Sequences'( Anchored Element Generator) of component 'Count Filter' (de.uni_koeln.spinfo.tesla.component.filter.CountFilter)
• Role 'Reference'( Anchored Element Generator) of component 'Count Filter' (de.uni_koeln.spinfo.tesla.component.filter.CountFilter)
• Role 'Words'( Tokenizer) of component 'Random Walk' (de.uni_koeln.spinfo.tesla.component.corpusstatistics.RandomWalkComponent)
• Role 'Items'( Anchored Element Generator) of component 'Structure Trimmer' (de.uni_koeln.spinfo.tesla.component.sequencer.StructureTrimmer)
• Role 'Structures'( Anchored Element Generator) of component 'Structure Trimmer' (de.uni_koeln.spinfo.tesla.component.sequencer.StructureTrimmer)
• Role 'Tokens'( Tokenizer) of component 'Berkeley Parser' (de.uni_koeln.spinfo.tesla.component.berkeleyparser.BerkeleyParserWrapper)
• Role 'Sequences'( Anchored Element Generator) of component 'Berkeley Parser' (de.uni_koeln.spinfo.tesla.component.berkeleyparser.BerkeleyParserWrapper)
• Role 'Test Data'( Annotator) of component 'JUnit Tests' (de.uni_koeln.spinfo.tesla.components.componenttester.ComponentTester)
• Role 'Items'( Anchored Element Generator) of component 'Sequence Length Filter' (de.uni_koeln.spinfo.tesla.component.filter.SequenceLengthFilter)
• Role 'Sequences'( Anchored Element Generator) of component 'Sequence Length Filter' (de.uni_koeln.spinfo.tesla.component.filter.SequenceLengthFilter)
• Role 'Reference'( Anchored Element Generator) of component 'Sequence Length Filter' (de.uni_koeln.spinfo.tesla.component.filter.SequenceLengthFilter)
• Role 'Sequences'( Anchored Element Generator) of component 'ABL Writer' (de.uni_koeln.spinfo.tesla.component.abl.AblWriterComponent)
• Role 'Sequence Items'( Anchored Element Generator) of component 'ABL Writer' (de.uni_koeln.spinfo.tesla.component.abl.AblWriterComponent)
• Role 'Annotations'( Annotator) of component 'Subclass Filter' (de.uni_koeln.spinfo.tesla.component.filter.SubclassFilter)
• Role 'Words'( Tokenizer) of component 'Coincidence Statistics' (de.uni_koeln.spinfo.tesla.component.corpusstatistics.CoincidenceStatisticsComponent)
• Role 'Corpus Words'( Tokenizer) of component 'Document Log Likelihood' (de.uni_koeln.spinfo.tesla.component.statistics.DocumentLogLikelihoodCalculator)
• Role 'Gold'( Annotator) of component 'Range Comparator' (org.schwiebert.tesla.rangecomparator.FastComparator)
• Role 'Validatable'( Annotator) of component 'Range Comparator' (org.schwiebert.tesla.rangecomparator.FastComparator)
• Role 'Annotations'( Anchored Element Generator) of component 'Document/Term Frequency Counter' (de.uni_koeln.spinfo.tesla.component.statistics.TfIdfCalculator)
• Role 'Sequences'( Anchored Element Generator) of component 'Suffix Export' (de.uni_koeln.spinfo.tesla.component.paradigms.select.CopyOfExportExchangables)
• Role 'Sequence Items'( Anchored Element Generator) of component 'Suffix Export' (de.uni_koeln.spinfo.tesla.component.paradigms.select.CopyOfExportExchangables)
• Role 'Words'( Tokenizer) of component 'Geographic Location Extender' (de.uni_koeln.spinfo.tesla.component.geolocations.GeoNamesExtender)
• Role 'Sequences'( Anchored Element Generator) of component 'ABL Select' (de.uni_koeln.spinfo.tesla.component.abl.AblSelectComponent)
• Role 'Sequence Items'( Anchored Element Generator) of component 'ABL Select' (de.uni_koeln.spinfo.tesla.component.abl.AblSelectComponent)
• Role 'Base Sequence'( Anchored Element Generator) of component 'Structure Combiner' (de.uni_koeln.spinfo.tesla.component.sequencer.StructureCombiner)
• Role 'New Items'( Anchored Element Generator) of component 'Structure Combiner' (de.uni_koeln.spinfo.tesla.component.sequencer.StructureCombiner)
• Role 'Sequence'( Anchored Element Generator) of component 'Filter Rewriter' (de.uni_koeln.spinfo.tesla.component.filter.FilterWriter)
• Role 'Sentences'( Anchored Element Generator) of component 'Time!' (de.uni_koeln.spinfo.tesla.component.time.TimeComponent)
• Role 'Words'( Anchored Element Generator) of component 'Time!' (de.uni_koeln.spinfo.tesla.component.time.TimeComponent)
• Role 'To Parse'( Anchored Element Generator) of component 'Time!' (de.uni_koeln.spinfo.tesla.component.time.TimeComponent)
• Role 'Items'( Sequence Annotator) of component 'Jython Porter Stemmer' (de.uni_koeln.spinfo.tesla.component.jython.indexer.JythonIndexer)
• Role 'Tokens'( Tokenizer) of component 'Stanford Named Entity Recognizer' (de.uni_koeln.spinfo.tesla.component.stanfordner.NamedEntityRecognizer)
• Role 'Reference'( Anchored Element Generator) of component 'Category Filter' (de.uni_koeln.spinfo.tesla.component.filter.CategoryFilter)
• Role 'Words'( Tokenizer) of component 'Morphemizer' (de.uni_koeln.spinfo.formanalysis.teslacomponents.SimpleMorphemizerComponent)
• Role 'Words'( Tokenizer) of component 'Token rewriter' (de.uni_koeln.spinfo.tesla.component.sequencer.TokenRewriter)
• Role 'Words'( Tokenizer) of component 'Gazetteer' (de.uni_koeln.spinfo.tesla.component.gazetteer.GazetteerComponent)