2
Import von SQL Server konvertiert, Datentypen nicht ordnungsgemäß konvertiert Stack: Installierte HDP-2.3.2.0-2950 Ambari mit 2,1Import von SQL Server, Datentypen nicht richtig
Ziel:
- Import Tabellen von SQL Server auf HDFS in Avro Format
- AUSSEN Hive Avro (SerDe) Tabellen erstellen, die die Daten aller haben
- AUSSEN Hive ORC Tabellen erstellen und einfügen in ORC select * from Avro Tabellen
- Tropfen die Avro Tabellen und führen Tests an den ORC Tabellen
Eine der Tabellen:
ECU_DTC_ID int
DTC_CDE nchar(20)
ECU_NAME nvarchar(15)
ECU_FAMILY_NAME nvarchar(15)
DTC_DESC nvarchar(MAX)
INSERTED_BY nvarchar(64)
INSERTION_DATE datetime
DTC_CDE_DECIMAL int
Wenn ich den normalen Sqoop Import ausführen, wird das zu konvertierende Zeitangabe lang, nchar und nvarchar zu Schnur. Die sich ergebende AVSC Datei wird wie gezeigt, wenn ich einen Hive Avro Tabelle zu erstellen, ist es nicht die generierten Avro-Dateien enthalten, so dass hinter einem leeren Tisch verlassen:
{
"type" : "record",
"name" : "DimECUDTCCode",
"doc" : "Sqoop import of DimECUDTCCode",
"fields" : [ {
"name" : "ECU_DTC_ID",
"type" : [ "null", "int" ],
"default" : null,
"columnName" : "ECU_DTC_ID",
"sqlType" : "4"
}, {
"name" : "DTC_CDE",
"type" : [ "null", "string" ],
"default" : null,
"columnName" : "DTC_CDE",
"sqlType" : "-15"
}, {
"name" : "ECU_NAME",
"type" : [ "null", "string" ],
"default" : null,
"columnName" : "ECU_NAME",
"sqlType" : "-9"
}, {
"name" : "ECU_FAMILY_NAME",
"type" : [ "null", "string" ],
"default" : null,
"columnName" : "ECU_FAMILY_NAME",
"sqlType" : "-9"
}, {
"name" : "DTC_DESC",
"type" : [ "null", "string" ],
"default" : null,
"columnName" : "DTC_DESC",
"sqlType" : "-9"
}, {
"name" : "INSERTED_BY",
"type" : [ "null", "string" ],
"default" : null,
"columnName" : "INSERTED_BY",
"sqlType" : "-9"
}, {
"name" : "INSERTION_DATE",
"type" : [ "null", "long" ],
"default" : null,
"columnName" : "INSERTION_DATE",
"sqlType" : "93"
}, {
"name" : "DTC_CDE_DECIMAL",
"type" : [ "null", "int" ],
"default" : null,
"columnName" : "DTC_CDE_DECIMAL",
"sqlType" : "4"
} ],
"tableName" : "DimECUDTCCode"
entschied ich mich --map-Spalte schließen -java:
sqoop import --connect 'jdbc:sqlserver://somedbserver;database=somedb' --username someusername--password somepassword --as-avrodatafile --num-mappers 8 --table DimECUDTCCode --map-column-java DTC_CDE=string,ECU_NAME=string,ECU_FAMILY_NAME=string,DTC_DESC=string,INSERTED_BY=string,INSERTION_DATE=timestamp --warehouse-dir /dataload/tohdfs/reio/odpdw/may2016 --verbose
aber ich bekomme die folgende Fehlermeldung:
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type timestamp
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string
16/05/12 09:43:12 ERROR tool.ImportTool: Imported Failed: No ResultSet method for Java type string
[[email protected] root]$
Was bin ich?
können Sie '--map-column-hive' versuchen und SQL Server-Spalten direkt den Hive-Spalten zuordnen. –
Aber warum Bienenstock, möchte ich Java verwenden, die nicht erfolgreich ist –
Ja, Sie sollten versuchen, Problem zu finden mit '--map-column-java'. Ich habe gerade eine Alternative gegeben, wenn Sie stecken geblieben sind, weil ich 'map-column-hive' versucht habe und es funktioniert hat. –