Hadoop

Create a reliable backup. Make sure you always have dependable copies of your data so you can restore your entire system or individual files.

Question by

sunhux

2018-10-09Solved

Comparing Splunk, ELK & Metron Hadoop

We are considering Splunk, ELK or Apache Metro Hadoop for SIEM.

Q1:
I've encountered nightmares with a top-end SIEM in the past when
querying/retrieving data : takes days & even crash : which of the
above has excellent super-speed log management & querying?

Q2:
I was told by an ex-colleague that Arcsight/Splunk requires CEF
(Common Event Format or syslog format) to be piped to them
as they can't accept any other format. A vendor using QRadar
told me QRadar requires syslog/CEF format inputs too.
I've SNMP traps / MIBS events (eg: from Cisco & proprietary
devices) that my ex-colleague told me can't be accepted by
Splunk/Arcsight, so would like to know if any of the 3 above
tools are more readily able to accept other SNMP/other
event formats

Q3:
Heard that ELK lacks policies which in the long run will be
costlier if we get consultants to customize : do the other
2 products have this concern.
Also, Splunk Enterprise goes by amount of logs & we're
concerned that too much logs (can be 500MB/month)
will make the cost high: weighing between customization
/set-up PS efforts & licensing costs based on amount of
logs (which I guess we can archive off older logs to reduce
the license cost), which of the 3 are more cost-effective?

4 Comments

Question by

MIHIR KAR

2018-07-30

Oracle to Hadoop HDFS and vice versa!

Hi Expert,

Could anybody please guide me how to load data from Oracle DB to Hadoop HDFS and the result back to Oracle DB.

Thanks in Advance!

4 Comments

Issue with high number of TCP CLOSE_WAIT socket connections on Hortonwork(HDP2.6.4) NameNodes & Metastore Server.
We frequently have very high number of CLOSE_WAIT socket connections on hadoop servers, as a result hadoop services are unavailable on Namenode servers. This happen after heavy ingestion of data in cluster. As a result, I need to restart the cluster after re-booting concerned servers.
I tried re-setting value of several TCP attributes on the servers, but this had not solve the problem.
Using lsop | grep CLOSE_WAIT, I can identified concerned processes which had CLOSE_WAIT socket connections, I killed the concerned process & try to re-start hadoop services but this had also not solve the problem.
I had monitored the servers for number of CLOSE_WAIT socket connections & whenever number of these keep rising , it's point to symptom that the hadoop services on NameNode are going to down in couple of minutes.
Any idea to solve this issue is welcome.

So i have a user schema like this:

var user_schema = new Schema({
   username:{type:String,required: true, unique : true, trim: true},
   college:{type:String,required: true},
   password:{type:String,required: true, trim: true},
   email:{type:String,required: true, unique : true, trim: true},
   phone:{type:Number,required: true, unique : true, trim: true},
   dp:{type:String},
   tag:{type:String},
   description:{type:String},
   friends:[{type:String}],
   pending:[{type:String}],
   skills:{type:String},
   bucket:[{type:String}]
  });

and my objective is, to search the all the documents in the collection to get people based on the following conditions:

1. They should not be in the users' "friends" array.
2. They should not be in the users' "pending" array.
3. They should have the same "tag" (a string value) as the user.

So, basically I have to compare the users' fields ("friends","pending" and "tags"), with fields of all documents in the whole collection.

How do I do it, using mongoose (nodejs mongodb library)

0 Comments

My requirement is to store multiple data types in same column of a hive table.
And also be able to read that data .e.g. if one records has array value for that column and another record has struct value or string , I may be able to fetch that value accordingly.

I could store json data as avro in hive table with data type of string for that particular column( since spark/sqlcontext inferred the data type as string for multiple data types in the same column).I am not able to operat on that data.I just can simply read by select columnname from table.

I have tried to use uniontype to load that column-table data(string) into another table with uniontype as data type for that column but it erred saying mismatch.

Error while compiling statement: FAILED: SemanticException [Error 10044]: line 18:36
Cannot insert into target table because column number/types are different 'test_uniontyp':
Cannot convert column 1 from string to uniontype<array<string>, string,struct<abd:array<struct<ax:string,bx:string>>>>
.

Any suggestion.

0 Comments

Question by

marrowyung

2018-06-13Solved

Hadoop for MySQL

hi,

how can MySQL work with / load and save data from Hadoop ? any build in tools for it ?

is it scalable solution ?

9 Comments

hi,

any Hadoop to DB2 gateway/proxy scaleoutable solution for DB2?

8 Comments

Question by

marrowyung

2018-05-16Solved

DB2 and Hadoop.

hi,

any product from DB2 can get data from Hadoop and store in a structured format?

3 Comments

hi,

any oracle product that help to transfer data in and out from Hadoop by using single PL SQL language ?

and also can do parallel data processing for that feature ?

7 Comments

Redefine Your Security with AI & Machine Learning

The implications of AI and machine learning in cyber security are massive and constantly growing, creating both efficiencies and new challenges across the board. Check out our on-demand webinar to learn more about how AI can help your organization!

Question by

marrowyung

2018-05-02Solved

polybase on MSSQL

hi,

anyone use polybase on MSSQL for hadoop ? is scale out feature of Polybase working fine? load balancing working well ?

16 Comments

Question by

marrowyung

2018-04-25Solved

MariaDB and mongo DB

hi,

anyone know how to intergrate MariaDB and Mongo DB so that they work together well ?

how about MariaDB and hadoop?

13 Comments

Question by

marrowyung

2018-04-25Solved

MSSQL and Mongo DB work together,

hi,

anyone know how to intergrate MS SQL and Mongo DB so that they work together well ?

how about MS SQL and hadoop?

4 Comments

LVL 9

Question by

Evan Cutler

2018-04-20Solved

What is the recommend SOLR architecture for a 180mil record dataset.

I have been asked to stand up a weighted search appliance for a company. The decision was to use SOLR to create the search tool so they can use the associated REST API for searches and recommendations.

I'm am still beginning in SOLR and have to ask a basic architecture question. I have a table with 220 elements, 130 Million record strong. I grow 5 million a year.

Does this become a Hadoop solution? or can this still be done with a single SOLR engine? I need to know which direction to start with so I do this right

Thanks much.

2 Comments

There is a partitioned(batch_date is partition) hive table(Table 1) with 3 columns and 1 partition

I'm trying to execute INSERT INTO TABLE PARTITION(batch_date='2018-02-22') select column 1, column 2, column 3 from Table 1 where column 1 = "ABC";

It returns zero records and in hdfs it is creating 3 empty files.

Can you please suggest me a solution on how to prevent these small files getting created in hdfs??

Note: Before running the INSERT statements I have set the below hive properties.

set hive.exec.dynamic.partition=true;
set hive.exec.dynamic.partition.mode=nonstrict;

0 Comments

basically I would like to create a sample hello world program in Map Reduce and scala which can hit the job history server.

2 Comments

Question by

Member_2_7966113

2018-01-31Solved

How to store Query Results to Local Drive

Hello Experts,

The following Hive Script retrieves data from the hdfs dfs drive on hadoop from the directory '/user/hive/geography'

I would like to store the results on a local drive called /hadoop/hdfs'

Can someone please show me how to modify the script so that it doesn't retrieve and store the results of the query to 'user/hive/geography', but instead stores the results from the query to '/hadoop/hdfs' (or any local drive)

The script is as follows:

DROP TABLE IF EXISTS HiveSampleIn; 
CREATE EXTERNAL TABLE HiveSampleIn 
(
 anonid int,
 eprofileclass int,
 fueltypes STRING,
 acorn_category int,
 acorn_group STRING,
 acorn_type int,
 nuts4 STRING,
 lacode STRING,
 nuts1 STRING,
 gspgroup STRING,
 ldz STRING,
 gas_elec STRING,
 gas_tout STRING
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '10' STORED AS TEXTFILE LOCATION '/user/hive/geography'; 

DROP TABLE IF EXISTS HiveSampleOut; 
CREATE EXTERNAL TABLE HiveSampleOut 
(

acorn_category int,
acorn_categorycount int )

ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '10' STORED AS TEXTFILE LOCATION '/user/hive/geography';


INSERT OVERWRITE TABLE HiveSampleOut
Select 
   acorn_category,
   count(*) as acorn_categorycount 
FROM HiveSampleIn Group by acorn_category

Thanks

2 Comments

Hello Experts,

I have created the following Hadoop Hive Script.

The script is attempting to store the results of a query into the following location:

LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/sampleout/';

However, I keep on getting the following error:

FAILED: ParseException line 9:0 Failed to recognize predicate 'ROW'. Failed rule: 'identifier' in table or column identifier
18/01/30 16:08:06 [main]: ERROR ql.Driver: FAILED: ParseException line 9:0 Failed to recognize predicate 'ROW'. Failed rule: 'identifier' in table or column identifier
org.apache.hadoop.hive.ql.parse.ParseException: line 9:0 Failed to recognize predicate 'ROW'. Failed rule: 'identifier' in table or column identifier

The Hive script is as follows:

[code]DROP TABLE IF EXISTS geography;
CREATE EXTERNAL TABLE geography
(
 anonid INT,
 eprofileclass INT,
 fueltypes STRING,
 acorn_category INT,
 acorn_group STRING,
 acorn_type INT,
 nuts4 STRING,
 lacode STRING,
 nuts1 STRING,
 gspgroup STRING,
 ldz STRING,
 gas_elec STRING,
 gas_tout STRING
)
ROW FORMAT DELIMITED
 FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/sampleout/'
TBLPROPERTIES ("skip.header.line.count" = "1");

Create table acorn_category_frequency
 as
select acorn_category,
 count(*) as acorn_categorycount
from geography
group by acorn_category,
ROW FORMAT DELIMITED
 FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/sampleout/';

[/code]

Can someone please help figure out where I'm going wrong in the script?

Thanks

Techies, Can someone isolate where I'm dropping the ball on getting the regex matches I'm expecting in this dataflow? My goal is to move the matched versions over to 1 kafka topic and the unmatched over to another kafka topic. Attached is the client.csv test file

Here's what the data flow looks like--with the regex used in the ExtractText config. NiFi uses Java's version of regular expressions.

client.csv

7 Comments

IT Pros Agree: AI and Machine Learning Key

We’d all like to think our company’s data is well protected, but when you ask IT professionals they admit the data probably is not as safe as it could be.

Question by

Member_2_7966113

2018-01-29

Hadoop Hive Query not executing

Hello Experts,

I have created the following Hadoop Hive HQL script, however, I keep on getting the following error

FAILED: ParseException line 21:44 missing EOF at ',' near ')'
18/01/29 21:37:15 [main]: ERROR ql.Driver: FAILED: ParseException line 21:44 missing EOF at ',' near ')'

The script is as follows:

DROP TABLE IF EXISTS HiveSampleIn;
CREATE EXTERNAL TABLE HiveSampleIn
(
 anonid INT,
 eprofileclass INT,
 fueltypes STRING,
 acorn_category INT,
 acorn_group STRING,
 acorn_type INT,
 nuts4 STRING,
 lacode STRING,
 nuts1 STRING,
 gspgroup STRING,
 ldz STRING,
 gas_elec STRING,
 gas_tout STRING
)
ROW FORMAT DELIMITED
 FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/samplein/';
tblproperties ("skip.header.line.count"="2");

DROP TABLE IF EXISTS HiveSampleOut; 
CREATE EXTERNAL TABLE HiveSampleOut 
(    
    acorn_category int
    
) 
ROW FORMAT DELIMITED
 FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/sampleout/';

INSERT OVERWRITE TABLE HiveSampleOut
Select 
   acorn_category
   
FROM HiveSampleIn Group by acorn_category;

Any help in fixing this problem will be greatly appreciate.

Question by

Member_2_7966113

2018-01-29

Hadoop Hive Query Provide missing EOF Error

Hello Community,

The Hive script I have created keeps throwing the following error:

Time taken: 2.634 seconds
FAILED: ParseException line 17:2 missing EOF at 'COLUMN' near ')'
18/01/29 10:29:53 [main]: ERROR ql.Driver: FAILED: ParseException line 17:2 missing EOF at 'COLUMN' near ')'
org.apache.hadoop.hive.ql.parse.ParseException: line 17:2 missing EOF at 'COLUMN' near ')'

Can someone please take a look at the Hive script and let me know where I might be going wrong?

[code]DROP TABLE IF EXISTS HiveSampleIn; 
CREATE EXTERNAL TABLE HiveSampleIn 
(
 anonid int,
 eprofileclass int,
 fueltypes STRING,
 acorn_category int,
 acorn_group STRING,
 acorn_type int,
 nuts4 STRING,
 lacode STRING,
 nuts1 STRING,
 gspgroup STRING,
 ldz STRING,
 gas_elec STRING,
 gas_tout STRING
) COLUMN FORMAT DELIMITED FIELDS TERMINATED BY (',') LINES TERMINATED BY '10' STORED AS TEXTFILE LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/samplein/'; 

DROP TABLE IF EXISTS HiveSampleOut; 
CREATE EXTERNAL TABLE HiveSampleOut 
(    
    acorn_category int
    
) COLUMN FORMAT DELIMITED FIELDS TERMINATED BY (',') LINES TERMINATED BY '10' STORED AS TEXTFILE LOCATION 'wasb://adfgetstarted@geogstoreacct.blob.core.windows.net/sampleout/';

INSERT OVERWRITE TABLE HiveSampleOut
Select 
   acorn_category,
   count(*) as acorn_categorycount 

FROM HiveSampleIn Group by acorn_category

[/code]
Cheers

Carlton

5 Comments

Question by

Member_2_7966113

2018-01-29

Hadoop Hive File Query

Hello Experts,

I would like to run a query on the attached file, but I don't know what type information is on the file in order to run a query on it.

Can someone let me know how to determine the information included in the file?

Regards

Carlton
VANQ_TRIAD_COLLS_20180118

Question by

Member_2_7966113

2018-01-21Solved

Hadoop Query Not executing

Hello Community,

I have created my first hql code, see below and I can't get any data to appear.. I have recently installed Sandbox. The installation comes with a few sample databases. I'm using the database called sample_07 to guide me with my own .hql code.

My hql code is as follows:

CREATE EXTERNAL TABLE mysample
(
 code STRING,
 description STRING,
 total_emp INT,
 salary INT
)
ROW FORMAT DELIMITED
 FIELDS TERMINATED BY ','
STORED AS TEXTFILE
LOCATION '/root/music'
TBLPROPERTIES ("skip.header.line.count" = "1");

However, when I run the code using Zeppellin Notebook with the following code, I can see the tables, but no data appears

%jdbc(hive)
select * from mysample limit 14

However, when I run the same code, with using the sample database called sample_07 both the tables and data appear.

I'm sure there is something very simple that I'm missing.

Can someone please let me know where I'm going wrong?

3 Comments

Question by

Member_2_7966113

2018-01-17

Hadoop Hive ParseException Error Help

Hello Experts,

I have run a hql called samplehive.hql, see attached. However, the script fails with the following error:

FAILED: ParseException line 1:2 cannot recognize input near 'D' 'R' 'O'
18/01/17 20:46:46 [main]: ERROR ql.Driver: FAILED: ParseException line 1:2 cannot recognize input near 'D' 'R' 'O'
org.apache.hadoop.hive.ql.parse.ParseException: line 1:2 cannot recognize input near 'D' 'R' 'O'

I'm very new to Hadoop Hive, can someone take a look at the script and let me know where I'm going wrong

Thanks
samplehive.txt

Question by

mokkan

2017-12-07

YARN memory allocation

Hello,

I am new to Hadoop. I have a question regarding yarn memory allocation. If we have 16GB memory in cluster, we can have least 3 4GB cluster an keep 4 GB for other uses. If a job needs 10 GB RAM, would it use 3 containers or use one container and will start using the ram rest of the RAM ?