1. Zuhause
  2. Frage und Antwort
  3. Import von SQL Server, Datentypen nicht richtig

Import von SQL Server, Datentypen nicht richtig

2016-05-12 18 views
2

Import von SQL Server konvertiert, Datentypen nicht ordnungsgemäß konvertiert Stack: Installierte HDP-2.3.2.0-2950 Ambari mit 2,1Import von SQL Server, Datentypen nicht richtig

Ziel:

  • Import Tabellen von SQL Server auf HDFS in Avro Format
  • AUSSEN Hive Avro (SerDe) Tabellen erstellen, die die Daten aller
    • haben
  • AUSSEN Hive ORC Tabellen erstellen und einfügen in ORC select * from Avro Tabellen
  • Tropfen die Avro Tabellen und führen Tests an den ORC Tabellen

Eine der Tabellen:

ECU_DTC_ID   int 
DTC_CDE    nchar(20) 
ECU_NAME   nvarchar(15) 
ECU_FAMILY_NAME  nvarchar(15) 
DTC_DESC   nvarchar(MAX) 
INSERTED_BY   nvarchar(64) 
INSERTION_DATE  datetime 
DTC_CDE_DECIMAL  int

Wenn ich den normalen Sqoop Import ausführen, wird das zu konvertierende Zeitangabe lang, nchar und nvarchar zu Schnur. Die sich ergebende AVSC Datei wird wie gezeigt, wenn ich einen Hive Avro Tabelle zu erstellen, ist es nicht die generierten Avro-Dateien enthalten, so dass hinter einem leeren Tisch verlassen:

{ 
    "type" : "record", 
    "name" : "DimECUDTCCode", 
    "doc" : "Sqoop import of DimECUDTCCode", 
    "fields" : [ { 
    "name" : "ECU_DTC_ID", 
    "type" : [ "null", "int" ], 
    "default" : null, 
    "columnName" : "ECU_DTC_ID", 
    "sqlType" : "4" 
    }, { 
    "name" : "DTC_CDE", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "DTC_CDE", 
    "sqlType" : "-15" 
    }, { 
    "name" : "ECU_NAME", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "ECU_NAME", 
    "sqlType" : "-9" 
    }, { 
    "name" : "ECU_FAMILY_NAME", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "ECU_FAMILY_NAME", 
    "sqlType" : "-9" 
    }, { 
    "name" : "DTC_DESC", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "DTC_DESC", 
    "sqlType" : "-9" 
    }, { 
    "name" : "INSERTED_BY", 
    "type" : [ "null", "string" ], 
    "default" : null, 
    "columnName" : "INSERTED_BY", 
    "sqlType" : "-9" 
    }, { 
    "name" : "INSERTION_DATE", 
    "type" : [ "null", "long" ], 
    "default" : null, 
    "columnName" : "INSERTION_DATE", 
    "sqlType" : "93" 
    }, { 
    "name" : "DTC_CDE_DECIMAL", 
    "type" : [ "null", "int" ], 
    "default" : null, 
    "columnName" : "DTC_CDE_DECIMAL", 
    "sqlType" : "4" 
    } ], 
    "tableName" : "DimECUDTCCode"

entschied ich mich --map-Spalte schließen -java:

sqoop import --connect 'jdbc:sqlserver://somedbserver;database=somedb' --username someusername--password somepassword --as-avrodatafile --num-mappers 8 --table DimECUDTCCode --map-column-java DTC_CDE=string,ECU_NAME=string,ECU_FAMILY_NAME=string,DTC_DESC=string,INSERTED_BY=string,INSERTION_DATE=timestamp --warehouse-dir /dataload/tohdfs/reio/odpdw/may2016 --verbose

aber ich bekomme die folgende Fehlermeldung:

16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_NAME to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column ECU_FAMILY_NAME to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_DESC to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTED_BY to string 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type string 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column INSERTION_DATE to timestamp 
16/05/12 09:43:12 ERROR orm.ClassWriter: No ResultSet method for Java type timestamp 
16/05/12 09:43:12 INFO orm.ClassWriter: Overriding type of column DTC_CDE to string 
16/05/12 09:43:12 ERROR tool.ImportTool: Imported Failed: No ResultSet method for Java type string 
[[email protected] root]$

Was bin ich?

Quelle

2016-05-12 Kaliyug Antagonist

+0

können Sie '--map-column-hive' versuchen und SQL Server-Spalten direkt den Hive-Spalten zuordnen. –

+0

Aber warum Bienenstock, möchte ich Java verwenden, die nicht erfolgreich ist –

+0

Ja, Sie sollten versuchen, Problem zu finden mit '--map-column-java'. Ich habe gerade eine Alternative gegeben, wenn Sie stecken geblieben sind, weil ich 'map-column-hive' versucht habe und es funktioniert hat. –

Antwort

0

Es stellt sich heraus, dass STRING und String und string von SQOOP anders behandelt werden. Richtig ist String.

Quelle

2018-01-09 06:10:40 peeyush

 Verwandte Themen

  • Keine verwandten Themen^_^