why is my solr engine only pulling 10 rows

I am learning SOLR for the first time.  I am using SOLR 7.2.1 plus java 8.  I am currently following this page:

http://www.codewrecks.com/blog/index.php/2013/04/29/loading-data-from-sql-server-to-solr-with-a-data-import-handler/

my query is "select * from mytable" with a million records on it.
But when I activate the full load (http://localhost:8983/solr/#/offrs/dataimport//dataimport) only ten records come into the core.

This is what I built:

data.import.xml:

    <dataConfig>  
        <dataSource type="JdbcDataSource"
                driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
                url="jdbc:sqlserver://<<SERVER>>;databaseName=<<DATABASE>>;"
                user="<<USER>>"
                password="<<PASSWORD"
        /> 
                 
        <document name="MY_DOCUMENT">  
            <entity name="doc" query="SELECT * FROM MYTABLE">  
                <field column='Field1' name='field1' />
    			<field column='field2' name='field2' />
                ...
            </entity
        </document>
    <dataConfig>

Open in new window


Schema.XML

    <schema name='doc' version='1.1'>
    	<types>
    		<fieldtype name='string' class='solr.StrField' />
    		<fieldtype name='long' class='solr.LongPointField' />
    		<fieldtype name='integer' class='solr.IntPointField' />
    		<fieldtype name='float' class='solr.FloatPointField' />
    	</types>
    
    	<fields>
    		<field name='field1' type='string' />
    		<field name='field2' type='string' />
    		<field name='field3' type='string' />
    	</fields>
    
    	<uniqueKey>field1</uniqueKey>
    
    </schema>

Open in new window


Solrconfig.xml

    <config>
    	<luceneMatchVersion>6.0</luceneMatchVersion>
    	<requestHandler name='standard' class='solr.StandardRequestHandler' default='true' />
    	
    	<admin>
    		<defaultQuery>*:*</defaultQuery>
    	</admin>
    	
    	<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">     
     
        <lst name="defaults">    
            <str name="config">data.import.xml</str>  
        </lst>    
    </requestHandler>  
    </config>

Open in new window

Is there another setting I am needing?

Update #1:
When looking at the SQL Server Activity Monitor, I see the query coming form SOLR....however, it is saying suspended with this error: ASYNC_NETWORK_IO.  I don't know if this helps, but I'm putting in as much data as I can.


Thanks.
LVL 9
Evan CutlerVolunteer Chief Information OfficerAsked:
Who is Participating?
 
Mark WillsConnect With a Mentor Topic AdvisorCommented:
Well, a million rows will probably cause a few problems....

ASYNC_NETWORK_IO timeouts (waits) happen when SQL Server is waiting on the "requester" to consume the output that SQL Server has delivered, but is obviously not being received / "consumed" as quickly as it is being delivered.

Try selecting top 10 or adding a where clause to limit the results so you can at least isolate volume as the cause... Or limiting batchsizes...

Sorry I cant give a better answer for you, my knowledge of SOLR isnt great, and hope you find an expert who can help. But you cannot simply throw as much as you can at SQL without it having a chance to process.

What client are you using ? Can you control batchsize ?

But anyway, that is what an ASYNCH error is all about....
0
 
Evan CutlerVolunteer Chief Information OfficerAuthor Commented:
I appreciate it, but in this case I'm thinking SQL Server is waiting on SOLR.
I don't know why tho.
0
 
Mark WillsConnect With a Mentor Topic AdvisorCommented:
0
Cloud Class® Course: Python 3 Fundamentals

This course will teach participants about installing and configuring Python, syntax, importing, statements, types, strings, booleans, files, lists, tuples, comprehensions, functions, and classes.

 
Evan CutlerVolunteer Chief Information OfficerAuthor Commented:
Cool. How do ifix that?
0
 
Mark WillsTopic AdvisorCommented:
Depending on your client, break it into batches.

And I am still hoping a SOLR expert will come along :)
0
 
Mark WillsTopic AdvisorCommented:
Hi Evan,

Any progress ?
0
Question has a verified solution.

Are you are experiencing a similar issue? Get a personalized answer when you ask a related question.

Have a better answer? Share it in a comment.

All Courses

From novice to tech pro — start learning today.