why is my solr engine only pulling 10 rows

I am learning SOLR for the first time.  I am using SOLR 7.2.1 plus java 8.  I am currently following this page:

http://www.codewrecks.com/blog/index.php/2013/04/29/loading-data-from-sql-server-to-solr-with-a-data-import-handler/

my query is "select * from mytable" with a million records on it.
But when I activate the full load (http://localhost:8983/solr/#/offrs/dataimport//dataimport) only ten records come into the core.

This is what I built:

data.import.xml:

    <dataConfig>  
        <dataSource type="JdbcDataSource"
                driver="com.microsoft.sqlserver.jdbc.SQLServerDriver"
                url="jdbc:sqlserver://<<SERVER>>;databaseName=<<DATABASE>>;"
                user="<<USER>>"
                password="<<PASSWORD"
        /> 
                 
        <document name="MY_DOCUMENT">  
            <entity name="doc" query="SELECT * FROM MYTABLE">  
                <field column='Field1' name='field1' />
    			<field column='field2' name='field2' />
                ...
            </entity
        </document>
    <dataConfig>

Open in new window


Schema.XML

    <schema name='doc' version='1.1'>
    	<types>
    		<fieldtype name='string' class='solr.StrField' />
    		<fieldtype name='long' class='solr.LongPointField' />
    		<fieldtype name='integer' class='solr.IntPointField' />
    		<fieldtype name='float' class='solr.FloatPointField' />
    	</types>
    
    	<fields>
    		<field name='field1' type='string' />
    		<field name='field2' type='string' />
    		<field name='field3' type='string' />
    	</fields>
    
    	<uniqueKey>field1</uniqueKey>
    
    </schema>

Open in new window


Solrconfig.xml

    <config>
    	<luceneMatchVersion>6.0</luceneMatchVersion>
    	<requestHandler name='standard' class='solr.StandardRequestHandler' default='true' />
    	
    	<admin>
    		<defaultQuery>*:*</defaultQuery>
    	</admin>
    	
    	<requestHandler name="/dataimport" class="org.apache.solr.handler.dataimport.DataImportHandler">     
     
        <lst name="defaults">    
            <str name="config">data.import.xml</str>  
        </lst>    
    </requestHandler>  
    </config>

Open in new window

Is there another setting I am needing?

Update #1:
When looking at the SQL Server Activity Monitor, I see the query coming form SOLR....however, it is saying suspended with this error: ASYNC_NETWORK_IO.  I don't know if this helps, but I'm putting in as much data as I can.


Thanks.
LVL 9
Evan CutlerVolunteer Chief Information OfficerAsked:
Who is Participating?

[Product update] Infrastructure Analysis Tool is now available with Business Accounts.Learn More

x
I wear a lot of hats...

"The solutions and answers provided on Experts Exchange have been extremely helpful to me over the last few years. I wear a lot of hats - Developer, Database Administrator, Help Desk, etc., so I know a lot of things but not a lot about one thing. Experts Exchange gives me answers from people who do know a lot about one thing, in a easy to use platform." -Todd S.

Mark WillsTopic AdvisorCommented:
Well, a million rows will probably cause a few problems....

ASYNC_NETWORK_IO timeouts (waits) happen when SQL Server is waiting on the "requester" to consume the output that SQL Server has delivered, but is obviously not being received / "consumed" as quickly as it is being delivered.

Try selecting top 10 or adding a where clause to limit the results so you can at least isolate volume as the cause... Or limiting batchsizes...

Sorry I cant give a better answer for you, my knowledge of SOLR isnt great, and hope you find an expert who can help. But you cannot simply throw as much as you can at SQL without it having a chance to process.

What client are you using ? Can you control batchsize ?

But anyway, that is what an ASYNCH error is all about....
0

Experts Exchange Solution brought to you by

Your issues matter to us.

Facing a tech roadblock? Get the help and guidance you need from experienced professionals who care. Ask your question anytime, anywhere, with no hassle.

Start your 7-day free trial
Evan CutlerVolunteer Chief Information OfficerAuthor Commented:
I appreciate it, but in this case I'm thinking SQL Server is waiting on SOLR.
I don't know why tho.
0
Mark WillsTopic AdvisorCommented:
0
IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

Evan CutlerVolunteer Chief Information OfficerAuthor Commented:
Cool. How do ifix that?
0
Mark WillsTopic AdvisorCommented:
Depending on your client, break it into batches.

And I am still hoping a SOLR expert will come along :)
0
Mark WillsTopic AdvisorCommented:
Hi Evan,

Any progress ?
0
It's more than this solution.Get answers and train to solve all your tech problems - anytime, anywhere.Try it for free Edge Out The Competitionfor your dream job with proven skills and certifications.Get started today Stand Outas the employee with proven skills.Start learning today for free Move Your Career Forwardwith certification training in the latest technologies.Start your trial today
SOLR

From novice to tech pro — start learning today.