Package: bdpar 3.1.0

bdpar: Big Data Preprocessing Architecture

Provide a tool to easily build customized data flows to pre-process large volumes of information from different sources. To this end, 'bdpar' allows to (i) easily use and create new functionalities and (ii) develop new data source extractors according to the user needs. Additionally, the package provides by default a predefined data flow to extract and pre-process the most relevant information (tokens, dates, ... ) from some textual sources (SMS, Email, YouTube comments).

Authors:Miguel Ferreiro-Díaz [aut, cre], David Ruano-Ordás [aut, ctr], Tomás R. Cotos-Yañez [aut, ctr], José Ramón Méndez Reboredo [aut, ctr], University of Vigo [cph]

bdpar_3.1.0.tar.gz
bdpar_3.1.0.zip(r-4.7)bdpar_3.1.0.zip(r-4.6)bdpar_3.1.0.zip(r-4.5)
bdpar_3.1.0.tgz(r-4.6-any)bdpar_3.1.0.tgz(r-4.5-any)
bdpar_3.1.0.tar.gz(r-4.7-any)bdpar_3.1.0.tar.gz(r-4.6-any)
bdpar_3.1.0.tgz(r-4.6-emscripten)
manual.pdf |manual.html✨
DESCRIPTION |NEWS
card.svg |card.png
bdpar/json (API)

# Install 'bdpar' in R:
install.packages('bdpar', repos = c('https://miferreiro.r-universe.dev', 'https://cloud.r-project.org'))

Bug tracker:https://github.com/miferreiro/bdpar/issues

Datasets:
  • bdparData - Example of the content of the files to be preprocessed.
  • emojisData - Emojis codes and descriptions data.

On CRAN:

Conda:

custom-flowcustom-pipespreprocessingr6

5.31 score 8 stars 17 scripts 352 downloads 34 exports 7 dependencies

Last updated from:e92df857b0. Checks:9 OK. Indexed: yes.

TargetResultTimeFilesSyslog
linux-devel-x86_64OK177
source / vignettesOK228
linux-release-x86_64OK165
macos-release-arm64OK138
macos-oldrel-arm64OK178
windows-develOK118
windows-releaseOK112
windows-oldrelOK99
wasm-releaseOK111

Exports:%>|%AbbreviationPipeBdparbdpar.logbdpar.OptionsConnectionsContractionPipeDefaultPipelineDynamicPipelineExtractorEmlExtractorFactoryExtractorSmsExtractorYtbidFile2PipeFindEmojiPipeFindEmoticonPipeFindHashtagPipeFindUrlPipeFindUserNamePipeGenericPipeGenericPipelineGuessDatePipeGuessLanguagePipeInstanceInterjectionPipeMeasureLengthPipeResourceHandlerrunPipelineSlangPipeStopWordPipeStoreFileExtPipeTargetAssigningPipeTeeCSVPipeToLowerCasePipe

Dependencies:data.tabledigestjsonliteR6rlistXMLyaml

A Brief Introduction to bdpar
Abstract | Introduction and basics | Instance | Types of input data available by default | How to create a customized input data extractors. | Enabling a new Instance. | Pipe | Dependencies | Pipes available by default | (i) Pipes of basic functionality | File2Pipe | FindEmojiPipe | FindEmoticonPipe | FindHashtagPipe | FindUrlPipe | FindUserNamePipe | GuessDatePipe | GuessLanguagePipe | MeasureLengthPipe | StoreFileExtPipe | TargetAssigningPipe | TeeCSVPipe | ToLowerCasePipe | (ii) Pipes that access external files | AbbreviationPipe | ContractionPipe | InterjectionPipe | SlangPipe | StopWordPipe | How to create your customized Pipe | Flow of Pipes (pipelining proccess) | Flow of Pipes available by default | Create your own flow of Pipes | Operator | bdpar.Options | [resources] | [teeCSVPipe] | [youtube] | [cache] | [parallel] | [verbose] | Cache functionality | Parallelization
Rendered frombdpar.Rmdusingknitr::rmarkdown

Last update: 2023-12-12
Started: 2019-07-09

Basic example using bdpar package
Introduction | Example | Visualization | Execution
Rendered frombdparExample.Rmdusingknitr::rmarkdown

Last update: 2020-11-25
Started: 2020-02-20

Image processing example using bdpar package
Introduction | Example | Preparation | Extractor | Creation of pipes | Execution
Rendered frombdparExampleImage.Rmdusingknitr::rmarkdown

Last update: 2020-11-25
Started: 2020-11-25

Readme and manuals

Help Manual

Help pageTopics
Class to find and/or replace the abbreviations on the data field of an InstanceAbbreviationPipe
Class to manage the preprocess of the files throughout the flow of pipesBdpar
Write messages to the log at a given priority level using the custom bdpar logbdpar.log
Object to handle the keys/attributes/options common to all pipeline flowbdpar.Options
Example of the content of the files to be preprocessed.bdparData
Class to manage the connections with YouTubeConnections
Class to find and/or replace the contractions on the data field of a InstanceContractionPipe
Class implementing a default pipelining process.DefaultPipeline
Class implementing a dynamic pipelining processDynamicPipeline
Emojis codes and descriptions data.emojisData
Class to handle email files with eml extensionExtractorEml
Class to handle the creation of Instance typesExtractorFactory
Class to handle SMS files with tsms extensionExtractorSms
Class to handle comments of YouTube files with ytbid extensionExtractorYtbid
Class to obtain the source field of an InstanceFile2Pipe
Class to find and/or replace the emoji on the data field of an InstanceFindEmojiPipe
Class to find and/or remove the emoticons on the data field of an InstanceFindEmoticonPipe
Class to find and/or remove the hashtags on the data field of an InstanceFindHashtagPipe
Class to find and/or remove the URLs on the data field of an InstanceFindUrlPipe
Class to find and/or remove the users on the data field of an InstanceFindUserNamePipe
Abstract super class that handles the management of the PipesGenericPipe
Abstract super class implementing the pipelining processGenericPipeline
Class to obtain the date field of an InstanceGuessDatePipe
Class to guess the language of an InstanceGuessLanguagePipe
Abstract super class that handles the management of the InstancesInstance
Class to find and/or remove the interjections on the data field of an InstanceInterjectionPipe
Class to obtain the length of the data field of an InstanceMeasureLengthPipe
bdpar customized forward-pipe operator%>|% operator-pipe
Class that handles different types of resourcesResourceHandler
Initiates the pipelining processrunPipeline
Class to find and/or replace the slangs on the data field of an InstanceSlangPipe
Class to find and/or remove the stop words on the data field of an InstanceStopWordPipe
Class to get the file's extension field of an InstanceStoreFileExtPipe
Class to get the target field of the InstanceTargetAssigningPipe
Class to handle a CSV with the properties field of the preprocessed InstanceTeeCSVPipe
Class to convert the data field of an Instance to lower caseToLowerCasePipe