July 22, 2012
by Rafał Kuć
In the previous part of the two – parts series about ActionGenerator we showed how to develop and run your own ActionGenerator. Today, we want to show you what action generators are there for you to use out-of-the-box and we want to share some insights about the future of this project.
ActionGenerator for ElasticSearch
The ag-player-es project contain all the code specific to action generators for ElasticSearch. You can find two sinks implemented – one for sending simple queries to ElasticSearch and the other for indexing data. They both use ElasticSearch REST API, so no dependencies are needed for those to work. The other piece available in ag-player-es are three players configured and ready to use:
- SimpleEsPlayerMain – ActionGenerator for running random queries to ElasticSearch.
- DictionaryEsPlayerMain – ActionGenerator for running queries to ElasticSearch. Queries are generated using a provided dictionary.
- DictionaryDataEsPlayerMain – ActionGenerator for indexing data to ElasticSearch. Fields content is generated using provided dictionary.
ActionGenerator for Solr
Similar to ag-player-es one can expect that ag-player-solr will contain all the code specific to action generators for Apache Solr. In this project you can find two sinks implementation – one for sending queries to ElasticSearch and one for indexing data. The first one uses Solr HTTP API and the other one uses XML to index data to Solr. Similar to ag-player-es, no dependencies are needed, so you should be able to use action generator for Solr with all recent Solr versions. Apart from that, there are three players configured and ready for use:
- DictionaryDataSolrPlayerMain - ActionGenerator for indexing data to Apache Solr. Fields content is generated using provided dictionary.
- DictionarySolrPlayerMain - ActionGenerator for running queries to Apache Solr. Queries are generated using provided dictionary.
- RandomQueriesSolrPlayerMain - ActionGenerator for running random queries to Apache Solr.
Using ActionGenerator for ElasticSearch
Lets concentrate on players available for ElasticSearch in ActionGenerator. As we wrote above, you have three main players for ElasticSearch that can be used out of the box.
SimpleEsPlayerMain
The simplest of the three generators available. It lets you generate random queries to a given index and with the use of the given field name. In order to use that player, you need to provide the following parameters:
- ElasticSearch base URL
- ElasticSearch index name
- Name of the field queries should be run against
- Number of events that should be generated
For example, you could run the following command and have 1000 queries sent to name field of the documents index on your local ElasticSearch instance:
java -cp ag-player-es-0.1.0-withdeps.jar \
com.sematext.ag.es.SimpleEsPlayerMain http://localhost:9200/ documents text 1000
DictionaryEsPlayerMain
The second generator that enables you to run queries uses a dictionary to generate text of your queries. It is similar to the SimpleEsPlayerMain, except for dictionary usage. In order to use that player, you need to provide the same parameters as to the SimpleEsPlayerMain and one additional parameters:
For example, you could run the following command and have 1000 queries sent to name field of the documents index on your local ElasticSearch instance. Queries would be generated using dict.txt dictionary (each line containing different query string):
java -cp ag-player-es-0.1.0-withdeps.jar com.sematext.ag.es.DictionaryEsPlayerMain http://localhost:9200/ documents text 1000 dict.txt
DictionaryDataEsPlayerMain
The one and only player that enables you to index data to your ElasticSearch instance. Just as the player discussed above, DictionaryDataEsPlayerMain also works with the help of a dictionary. You need to provide the following parameters in order to use this player:
- ElasticSearch base URL
- ElasticSearch index name
- ElasticSearch type name
- Number of events that should be generated
- Dictionary path
- One or more fields and their types
So, for example if you would like to index 100.000 documents to documents index under document type to your local ElasticSearch instance you could run the following
java -cp ag-player-es-0.1.0-withdeps.jar com.sematext.ag.es.DictionaryDataEsPlayerMain http://localhost:9200/ documents document 100000 dict.txt id:numeric title:text likes:numeric
Right now, the following field types are available:
We plan to add more types in the future. Pull requests with patches are welcome!
Using ActionGenerator for Apache Solr
Players available for Apache Solr are similar to the ones available for ElasticSearch, but lets quickly look at them, too.
RandomQueriesSolrPlayerMain
This player is similar to the SimpleEsPlayerMain – it also generates random queries, but to Apache Solr. In order to use it, you need to provide the following parameters:
- Apache Solr core search handler URL
- The field queries should be run against
- Number of queries to be generated
For example, you could run the following command and have 1000 queries sent to name field of the documents core of your local Apache Solr instance:
java -cp ag-player-solr-0.1.0-withdeps.jar com.sematext.ag.solr.RandomQueriesSolrPlayerMain http://localhost:8983/solr/documents name 1000
DictionarySolrPlayerMain
DictonarySolrPlayerMain is similar to RandomQueriesSolrPlayerMain except it uses dictionary to generate queries. In order to use this player you need to provide one additional parameter compared to the ones you provided to DictionarySolrPlayerMain:
For example, you could run the following command and have 1000 queries sent to name field of the documents core of your local Apache Solr instance. Queries would be generated using dict.txt dictionary (each line containing different query string):
java -cp ag-player-solr-0.1.0-withdeps.jar com.sematext.ag.solr.DictionarySolrPlayerMain http://localhost:8983/solr/documents name 1000 dict.txt
DictionaryDataSolrPlayerMain
The last player I’d like to tell you about is the one that enables data indexation to your Apache Solr instance. DictionaryDataSolrPlayerMain is similar to its ElasticSearch counterpart. You need to provide the following parameters in order to use this player:
- Apache Solr core update handler URL
- Number of events that should be generated
- Dictionary path
- One or more fields and their types
For example, if you would like to index 100000 documents to documents core of your local Apache Solr instance you could run the following
java -cp ag-player-solr-0.1.0-withdeps.jar com.sematext.ag.solr.DictionaryDataSolrPlayerMain http://localhost:8983/solr/documents/update/ 100000 dict.txt id:numeric title:text likes:numeric
Lets omit the field types description as it was provided during DictionaryDataEsPlayerMain description above.
Calculating Metrics
Currently, the AbstractHttpSink class has the ability to gather metrics about how the system to which you are sending events is behaving. You can choose between two methods of metrics output – to the standard output or to a file. To enable metrics tracking you need to pass the -DenableMetrics=true parameter when running ActionGenerator. This parameter enables metrics gathering and outputs those to standard output. In order to change that behavior and output the metrics to a file you need to pass the -DmetricsType=file parameter. In addition, you need to specify which directory the metrics should be written to – you do that by passing -DmetricsDir=/path/to/output/dir/ parameter with the value of the directory. Please remember that the directory needs to be created before running your action generator.
Maven Artifacts
The libraries for all the projects creating ActionGenerator can be found in the Sonatype maven repository (
http://oss.sonatype.org/content/repositories/releases/
) under the following dependencies:
Action Generator
<dependency>
<groupId>com.sematext.ag</groupId>
<artifactId>ag-player</artifactId>
<version>0.1.0</version>
</dependency>
Action Generator for ElasticSearch
<dependency>
<groupId>com.sematext.ag</groupId>
<artifactId>ag-player-es</artifactId>
<version>0.1.0</version>
</dependency>
Action Generator for Solr
<dependency>
<groupId>com.sematext.ag</groupId>
<artifactId>ag-player-solr</artifactId>
<version>0.1.0</version>
</dependency>
Plans for the Future
We plan to release action generators for SenseiDB, as well as expand the number of sinks and players ready to be used out of the box. Of course, as always, patches are welcome and if you find any problems with ActionGenerator or if you identify missing features, please open an issue.
