Data import help needed

English Forum Technical
#1

Hello

I am trying to get to grips with the data import procedure, but the only coherent information I can find is in the Technical Documentation library under « CSV Datasource ». It describes how to create an XML file for the data mapping, which I get the gist of, but I have no idea of where to go from there. Where should this XML file be, for a start? I looked at a system with demo data, in search of a clue, and I can see that there is such a file under [module]/src/main/resources/data-init/input/files - does the import program automatically look in this directory, or does this need to be configured somewhere? In the same directory is a CSV, in this case, Lead.csv file with just the header row. Do I need to do the same for a different import? And once I have done this, I presume I need to access it by adding a new record in the Technical Maintenance->Imports form - again, I have no idea how to do this in such a way that it finds the right mapping file in the right module. If anyone who has set up an import could find a moment to give a step-by-step guide to how to do it, I would be immensely grateful - Tom

#2

I’m also struggling with data import as the documentation is not well written and lack of good examples. I’m getting hard time to import entities with OneToMany relationship.
The data-init folder is for initializing the database the first time you install Axelor (when the database is empty). If you build a new version and deploy it on an existing database, only new entities get initialized.
For the data in the demo directory, they are loaded only when you explicitly ask for that (by setting data.import.demo-data to true in the application.properties or sometimes in the process of installing/configuring a new module).
In general, if you want to import some data, you will probably need to use the Maintenance/Import tool, but for that, you have to prepare the xml configuration file and the csv data file by yourself, which could be sometimes tricky, especially when the data is related somehow to other data (and this is by the way the part that is not well documented)

#3

Hi

when I made, for a customer, the massive import data I found this work in this way very useful:

  • locking how the demo import of the object worked (for example, city)
  • getting the csv and getting the complete schema used
  • identify what xml describe the data model (for example for city was base-import.xml)
  • using the batch to load data to make some test

In this why you understand how it work and how the schema used and after that it will be easy
generate the correct header for importing what you need

(we imported city, region, dept etc for italy, all the customer, all the contact and all the connection between them)

#4

Hi,
can you take a look at my post CSV import Problem and tell me what I’m doing wrong
Regards

#5

Hello Davide

I guess that where I am stuck is with not understanding how the Maintenance->Imports program works (I can find no documentation on this). Looking at this again, I am guessing that perhaps I create the required xml and csv files on my local PC, and then use the « upload » buttons in this program to copy them to some fixed destination within Axelor? And if so, does the csv file contain both the column headings and the data rows? And if that is correct, then I subsequently just click the Import button? And then, how do I effect subsequent imports using the same rules - do I upload a new csv file and click Import again? It looks like you have worked out how to do this, so if you have time to pass on your experience, I (and maybe other users) would be very grateful.

Regards, Tom

#6

Have used import quite extensively and share a bit

  1. Always include a importID, because you may need to update the same record. This will solve your problem of « subsequent imports ».
  2. When you import, you may find errors. Learn to use the log to check. Once there is error, axelor will not import the data. Log is your best friend over here.
  3. For your case of HR subcontract errors, see whether the system found records.

From your other thread of issues, you look quite experienced already and wth @bDurtaut 's help, should be fine by now?

Ignore if my reply sounds amateurish. Thanks.

#7

Thankyou for the tips, Andrew - much appreciated. But what I am failing to grasp is not (at least not yet) the syntax and guidelines of the load script so much as the basic steps of how to put it all in place in conjunction with the Maintenance->Imports program. Like I said, do I simply create the XML file on a local PC, and the upload button will magically copy it to the right place? And with the CSV file, do I upload a new CSV file each time I want to effect an import? And does every new CSV file need the header row as well as the data rows? Is the location of the XML and CSV files automatically excluded from (Git) version control, and if not, can that location be identified so that I can exclude it? I know that sounds like a load of dumb questions, but the procedure is far from obvious to me!

#8

Hi @tglare
Noted and just sharing what I see.

  1. I used the local install on my windows computer. My CSV and XMLis loaded into var>data>upload
    image. This should be similiar for online version.

  2. I created multiple import files with departments.

    image
    image1318×492 56.9 KB

So I guess the above 2 will map the file to this folder.

  1. I only upload those item that need changes, by using the importID.
    (I wouldn’t know how to to do it otherwise, which is a dangerous point. Because if you create a record in the system and export, there is no « Export ID », then I may a issue. Have tried exporting and it shows a blank field.Still figuring out.
    Odoo, for example, auto-assign a ID called export_ID for such case. This allows correct referencing).

  2. It looks like a generic path for storage and guess git version control should ignore this path.

So if you have access to the server backend, you may want to verify var>data>upload path.

Cheers.