Data Import Module

I've been working on a module with some advanced tools for importing data into Gibbon. Our school was previously using Blackbaud's Education Edge/Faculty Access and just started using Gibbon this year, so it was important to migrate thousands of various records into the new system and help ensure that the data is formatted and validated before it goes into the database.

If anyone is interested in using or testing this module you can find the repository on GitHub:
https://github.com/SKuipers/module-dataAdmin/tree/master/Data Admin

Download here: https://github.com/SKuipers/module-dataAdmin/archive/master.zip

Install in the modules folder & activate in System Admin to use. Works with v12 and v13(dev).

Features: (v1.0)
• A multi-part import form for improved control over incoming data
• Performs a dry-run before importing to see the anticipated results
• Handles relational fields (eg: transforms usernames into gibbonPersonID on import)
• Filters and validates many types of data (dates, emails, urls, roles, etc)
• Assign columns to database values ad-hoc and can skip non-required columns
• Choose to Update & Insert, Update Only or Insert Only when importing
• Create a database snapshot before importing to rollback changes if needed
• Keeps import logs tracking the user who made them and the results of the import
• Remembers the column order from the last import to speed up future imports
• Choose to export an excel file with all importable columns pre-filled
• Imports defined with a flexible YML syntax

How to use:
Once Installed, go to Admin > Data Admin. This shows a list of all the types of imports currently setup (new ones will continue to be added, feel free to request something if it's not in the list). The difference between these and regular Gibbon imports is they must have a header line in the csv file. The headers come into play once you go to step 2 where you can assign columns to fields, and optionally skip columns. Step 3 is a dry-run so you can see the results and any data formatting errors before the database is changed. If everything looks good, step 4 will complete the import into your database.

Snapshots are useful as an extra layer of database backup. Before you perform an import you can take a snapshot and if things go horribly wrong you can restore the snapshot. These aren't meant for live systems where people might be making changes continuously, or as a standalone backup method (they store locally in the upload folder, where as a good backup method should be on a different drive).

To help with formatting spreadsheets into csv files you can open an import type and click Export Structure. This will give you a spreadsheet with pre-filled column names. You can also click Export Data to make changes to a set of data in spreadsheet form then re-import it. To do this use "Update Only" and "Same as below (skip first column)" in Step 1, and the "Use database ID field" option in Step 2. Both Export options create an .xls file which would need saved as .csv to re-import.

I recommend using it on a local or development system first. I have tested and used it quite a bit myself however there are no guarantees. This program is distributed in the hope that it will be useful, but without any warranty; without even the implied warranty of merchantability or fitness for a particular purpose. See the GNU General Public License for more details.

Feel free to post any questions or send feedback, either in this thread or email me: sandra.kuipers [at] tis.edu.mo

Thanks!
admin

Comments

  • Sandra,

    This looks like some amazing work. Thanks for releasing it under a GPL license, as this kind of community involvement will definitely help Gibbon grow (I know it contains some Gibbon core code, and so must be under a GPL license, but I know you could have coded it from scratch and made it proprietary).

    I don't have the cause (or time!) to test this out right now, but I know your work is always meticulous, so I would really like to host it on the Extend page of gibbonedu.org. Would that be OK by you? I will list you as creator (do you want me to link you to a particular website?).

    Let me know.

    Thanks,

    Ross
  • Sure, I'll polish up a couple more things and let you know. Aside from the starter module stuff it is pretty much coded from scratch, but I'm happy to share it with the community : )

    Thanks!
    adminjavierlgc
  • Wow, thanks skuipers...wish I had this a few weeks ago...keep it up!

    admin
  • Hiya, might be being an idiot, but none of the 'export data structures' work to produce an xls for me?

    e.g.

    The web page at http://[domain]/modules/Data Admin/export_run.php?type=studentNotes might be temporarily down or it may have moved permanently
    admin
  • Ah scratch that, extensions missing :)
  • BeardyDude, welcome to the Gibbon community! We definitely welcome anyone who suspects they "might be being an idiot" ; ) We've all be there. Glad that you got the issue sorted.
    flygye12
  • Hi all,

    I've released v1.2.03 of Data Admin (download here, or on GitHub here), which adds a tool called Combine Similar Fields to help with duplicate user data, especially useful when importing data from other systems but also if duplicate data has accumulated over time.

    Combine Similar Fields: With user-entered data it's common to end up with a variety of details that all mean the same thing or are spelled incorrectly. This tool helps find and combine those fields. This can be particularly helpful if you're generating reports.


    Two modes: Assisted mode aims to help find matches between similar values (it can result in false positives though, which is also why there's manual mode):

    Manual mode lists all data and frequencies, letting you select and combine fields:

    In each mode you can select which fields to combine and re-name them as a group.

    More tools to help refine and merge duplicate data are in the works :smiley:
    admin
  • Also anyone working on migrating large data sets from other systems (before importing into Gibbon), be sure to check out Open Refine by Google
    adminjavedsahil
  • Sandra, Combine Similar Fields looks amazing. Will try and update our systems and have a go. Thanks!
    javedsahil
  • Sandra Buenas noches, felicitacion por tu esfuerzo de facilitar las cosas a los administradores de Gibon
    Tengo una pregunta por que cuando abro gibbon en español me aparecen algunos modulos como DATA ADMIN completaente en Ingles. Estuve revisando con la Traduccion de Poeditor y los terminos estan casi todos traducidos al español.
    En algunos otros modulo he encontrado en el menú algunos terminos en ingles y la mayoria en espeñol-

    Existe la posibilidad de convertir los términos que aún subsisten en Gibnon en Español, para que no quede una parte en ingles y otras en españoll?

    Hay que pedir algun codigo de acceso a cada uno de los archivos?

    Si te puedo colaborar en algo para terminar de hacer las traducciones estoy revisando en todo el sistema de POE la traduccion.

    Saludos
  • Hi Beatriz,

    Much of the functionality of Data Admin is now in the core under System Admin > Import from File. These should be translatable through the Gibbon project in POEditor. Is it just Data Admin, or other modules that you're looking to translate? Translating the additional non-core modules is something we've started to do in recent versions, but it's not setup for all the modules yet, it's something we're adding on a case-by-case basis, as time permits.
Sign In or Register to comment.