Tidying Motif Metadata

memes uses the universalmotif package to simplify working with motif metadata. universalmotif objects can be represented in an alternative form, the unviversalmotif_df which allows users to manipulate motif metadata just as they would a normal R data.frame (in fact, they are just R data.frames).

These objects are useful for tidying motif metadata to prepare a motif database for use with memes, or performing any other data-driven tasks involving motifs. Here I describe one way these data structures can be used to construct a motif database for use with memes.

The MotifDb package makes it easy to query thousands of motifs from public databases. Here, I will describe how to use one of these queries as input to memes functions, and how to manipulate the resulting motifs to prepare them for MEME suite tools.

I will use the motifs from the FlyFactorSurvey as an example. They can be accessed from MotifDb using the following query.

Use universalmotif::convert_motifs() to convert a MotifDb query into motif objects. In many cases, the resulting list can be used directly as input to memes functions, like runTomTom, or runAme.

But there are some issues with this database. For example, the following motif name is the FlyBase gene number, and the alternate name is the actual informative name of the PWM. The MEME Suite relies more heavily on the primary name, so it would be nice if the database used interpretable names.

The universalmotif function to_df() converts universalmotif lists into universalmotif_df format which can be used to update motif entries. This is particularly useful when dealing with several motifs at once.

The columns of the universalmotif_df can be freely changed to edit the properties of the motifs stored in the motif column. Just like standard data.frames, additional columns can be added to store additional metadata. For more details on these objects see the help page: ?universalmotif::to_df.

using the universalmotif_df, we can quickly see that the issue with FBgn numbers only applies to certain entries. And TFs which are represented by multiple motifs in the database are assigned the same name. The MEME Suite tools which use a motif database (like TomTom and AME) require that the entries have unique primary identifiers, therefore the default names will not be appropriate.

However, the altname slots from the motifDb query are already unique, so we can make them the primary name.

Next to solve the issue with the FBgn’s. FBgn numbers are unique identifiers for a gene within a given FlyBase reference assembly. However, FBgn numbers are not stable over time (i.e. the same gene may have a different FBgn number between reference assemblies), therefore they are unreliable values to determine the correct gene symbol. FlyBase.org has a nice conversion tool which can be used to update FBgn numbers.

As of this writing in March 2021, the FBgn entries provided by the Fly Factor Survey database are out of date. In order to demonstrate an example of methods for tidying motif metadata, I won’t use the FlyBase conversion tool, but will instead highlight some approaches which may be more generally useful when working with motif databases from disparate sources.

For this example, we will try to grab the correct gene name from the motif name, which is stored in the first field of the name, formatted as follows: “”.

We use tidyr::separate to split out the first entry to the tifd column, then only use this value if the altname contains an FBgn.

Next, because the FBgn’s are out of date, we will remove them from the “names” to shorten up the motif names. This also makes the motif name more comparable to the original motif names from the FlyFactor Survey.

A few reality checks

It’s worth taking a look at the instances where the altname and our parsed tfid do not match. This is a good way to ensure we haven’t missed any important edge cases in the data. As new edge cases are encountered, we can develop new rules for tidying the data to ensure a high quality set of motifs.

Start by simply filtering for all instance where there is a mismatch between altname and tfid.

Carefully compare the altname, name, and tfid columns. Why might the values differ? Are there instances that make you question the data?

flyFactor_data %>% 
  dplyr::filter(altname != tfid) %>% 
  # I'm only showing the first 5 rows for brevity, but take a look at the full
  # data and see what patterns you notice
  head(5)
#>           motif altname         name tfid      organism consensus alphabet
#> 1 * <mot:abd-A>   abd-A Abd-A_FlyReg  Abd Dmelanogaster  KNMATWAW      DNA
#> 2 * <mot:Abd-B>   Abd-B Abd-B_FlyReg  Abd Dmelanogaster  YCATAAAA      DNA
#> 3 * <mot:abd-A>   abd-A    AbdA_Cell AbdA Dmelanogaster   TTAATKA      DNA
#> 4 * <mot:abd-A>   abd-A  AbdA_SOLEXA AbdA Dmelanogaster  NTTAATKR      DNA
#> 5 * <mot:Abd-B>   Abd-B    AbdB_Cell AbdB Dmelanogaster GWTTTATKA      DNA
#>   strand  icscore nsites type pseudocount          bkg      dataSource
#> 1     +- 7.371641     37  PPM           1 0.25, 0..... FlyFactorSurvey
#> 2     +- 7.831193      7  PPM           1 0.25, 0..... FlyFactorSurvey
#> 3     +- 8.976569     18  PPM           1 0.25, 0..... FlyFactorSurvey
#> 4     +- 8.330000    662  PPM           1 0.25, 0..... FlyFactorSurvey
#> 5     +- 9.147674     21  PPM           1 0.25, 0..... FlyFactorSurvey
#> 
#> [Hidden empty columns: family, bkgsites, pval, qval, eval.]
#> [Rows marked with * are changed. Run update_motifs() or to_list() to
#>   apply changes.]

One thing that becomes obvious is that many motifs have mismatched altname/tfid values because of capitalization or hyphenation differences. You can use domain-specific knowledge to assess which one is correct. For Drosophila, “abd-A” is correct over “AbdA”, for example.

After manually inspecting these rows, I determined that instances of different capitalization, hyphenation, or names that contain “.” or “()” can be ignored. To further investigate the data, I will ignore capitalization and special character differences as follows:

flyFactor_data %>% 
  # calling tolower() on both columns removes capitalization as a difference
  dplyr::filter(tolower(altname) != tolower(tfid),
                # Select all altnames that do not contain "-", "." or "("
                !grepl("-|\\.|\\(", altname),
                ) %>% 
  # I'll visalize only these columns for brevity
  dplyr::select(altname, tfid, name, consensus) %>% 
  head(10)
#>    altname    tfid              name            consensus
#> 1       da      ac    ac_da_SANGER_5             RCACCTGC
#> 2       da    amos amos_da_SANGER_10          RMCAYMTGBCV
#> 3       da     ase  ase_da_SANGER_10              CACCTGY
#> 4       da     ato  ato_da_SANGER_10          MCAYMTGNCRC
#> 5       da     ato ato_da_SANGER_5_2            GNCAKRTGN
#> 6       da     ato ato_da_SANGER_5_3            MCAYMTGNC
#> 7       da    cato cato_da_SANGER_10           MRCANMTGWC
#> 8      Zif CG10267  CG10267_SANGER_5           WNYAACACTR
#> 9      Zif CG10267  CG10267_SOLEXA_5 CNNNNWAYAACACTASNMNN
#> 10    mamo CG11071  CG11071_SANGER_5         YMMGCCTAYNNM
#> 
#> [Note: incomplete universalmotif_df object.]

Next, what is obvious is that several altnames set to “da” have a high number of mismatched tfids. For instance, amos_da_SANGER_10. When checking the FlyFactorSurvey page for da, it reveals only 1 motif corresponds to this factor. Checking the page for amos shows a match to amos_da_SANGER_10. Therefore, we can conclude that factors assigned the name of da are incorrectly assigned, and we should prefer our parsed tfid.

flyFactor_data %<>% 
  # rename all "da" instances using their tfid value instead
  dplyr::mutate(altname = ifelse(altname == "da", tfid, altname))

Now we’ve handled the “da” mismatches, we filter them out to identify new special cases.

flyFactor_data %>% 
  dplyr::filter(tolower(altname) != tolower(tfid),
                !grepl("-|\\.|\\(", altname)) %>% 
  dplyr::select(altname, tfid, name, consensus) %>% 
  head(10)
#>    altname    tfid             name            consensus
#> 1      Zif CG10267 CG10267_SANGER_5           WNYAACACTR
#> 2      Zif CG10267 CG10267_SOLEXA_5 CNNNNWAYAACACTASNMNN
#> 3     mamo CG11071 CG11071_SANGER_5         YMMGCCTAYNNM
#> 4      lms CG13424     CG13424_Cell             NYTAATTR
#> 5      lms CG13424 CG13424_SOLEXA_2           NNYTAATTRN
#> 6      lms CG13424   CG13424_SOLEXA              YTAATTR
#> 7      erm CG31670 CG31670_SANGER_5          AAAWGMGCAWC
#> 8      erm CG31670 CG31670_SOLEXA_5      NNAAAWGAGCAAYNV
#> 9     Spps  CG5669 CG5669_SANGER_10      DRKGGGCGKGGCCAM
#> 10    Spps  CG5669  CG5669_SOLEXA_5      NGKGGGCGKGGCNWN
#> 
#> [Note: incomplete universalmotif_df object.]

The next thing to notice about these data is that entries with “CG” prefixed tfids are often mismatched. This is because when the FlyFactor survey was conducted, many genes were unnamed, and thus assigned a CG from FlyBase. As time has gone on, some CG’s have been named. Checking the FlyBase page for CG10267 reveals that it has been renamed “Zif”. This matches with the altname, so we conclude that rows with a “CG” tfid can be safely skipped as their altname contains the new gene symbol.

flyFactor_data %>% 
  dplyr::filter(tolower(altname) != tolower(tfid),
                !grepl("-|\\.|\\(", altname),
                # Remove CG genes from consideration
                !grepl("CG\\d+", tfid)
                ) %>% 
  dplyr::select(altname, tfid, name, consensus)
#>    altname tfid                 name   consensus
#> 1      cyc  Clk     Clk_cyc_SANGER_5    MCACGTGA
#> 2   CG6272  Crc  Crc_CG6272_SANGER_5 ATTACRTCABC
#> 3      Max   dm     dm_Max_SANGER_10  RNCACGTGGT
#> 4      Clk  gce     gce_Clk_SANGER_5    GCCACGTG
#> 5      Jra  kay     kay_Jra_SANGER_5 NATGASTCAYC
#> 6  schlank Lag1            Lag1_Cell  CYACYAAAWT
#> 7  schlank Lag1          Lag1_SOLEXA  CYACYAAWWT
#> 8     Lim1  lim         lim_SOLEXA_2   NNTAATTRV
#> 9      Mnt  Max     Max_Mnt_SANGER_5     CCACGTG
#> 10     Clk  Met     Met_Clk_SANGER_5    GACACGTG
#> 11  bigmax  Mio  Mio_bigmax_SANGER_5    ATCACGTG
#> 12     Bgb  run          run_Bgb_NBT   WAACCGCAR
#> 13     Clk  tai     tai_Clk_SANGER_5   RCACGTGTC
#> 14     cyc  tgo     tgo_cyc_SANGER_5   GTCACGTGM
#> 15     sim  tgo     tgo_sim_SANGER_5  GWACGTGACC
#> 16    sima  tgo    tgo_sima_SANGER_5   GTACGTGAC
#> 17      ss  tgo      tgo_ss_SANGER_5    TGCGTGAC
#> 18     tai  tgo     tgo_tai_SANGER_5   RCACGTGAC
#> 19     trh  tgo     tgo_trh_SANGER_5  RTACGTGACC
#> 20  CG6272 Xrp1 Xrp1_CG6272_SANGER_5  ATTRCRYMAY
#> 
#> [Note: incomplete universalmotif_df object.]

The remaining rows (only 20 values) can be manually inspected for any discrepancies. I went through each entry by hand, looking up their motifs on FlyFactor, and their gene names on FlyBase to determine the best way to handle these motifs. Sometimes the best way to be sure your data are high quality is to carefully inspect it!

I determined from this that a few altnames need swapping, and one motif I will remove because it is unusual (Bgb has an identical motif to run, but the motif is marked “run” on the FlyFactor website).

I’ll make those changes to the data:

swap_alt_id <- c("CG6272", "Clk", "Max", "Mnt", "Jra")
remove <- "Bgb"

flyFactor_data %<>% 
  dplyr::mutate(altname = ifelse(altname %in% swap_alt_id, tfid, altname)) %>% 
  dplyr::filter(!(altname %in% remove))

Finally, the remaining motif metadata is also OK based on my manual inspection.

flyFactor_data %>% 
  dplyr::filter(tolower(altname) != tolower(tfid),
                !grepl("-|\\.|\\(", altname),
                # Remove CG genes from consideration
                !grepl("CG\\d+", tfid)
                ) %>% 
  dplyr::select(altname, tfid, name, consensus)
#>    altname tfid                name  consensus
#> 1      cyc  Clk    Clk_cyc_SANGER_5   MCACGTGA
#> 2  schlank Lag1           Lag1_Cell CYACYAAAWT
#> 3  schlank Lag1         Lag1_SOLEXA CYACYAAWWT
#> 4     Lim1  lim        lim_SOLEXA_2  NNTAATTRV
#> 5   bigmax  Mio Mio_bigmax_SANGER_5   ATCACGTG
#> 6      cyc  tgo    tgo_cyc_SANGER_5  GTCACGTGM
#> 7      sim  tgo    tgo_sim_SANGER_5 GWACGTGACC
#> 8     sima  tgo   tgo_sima_SANGER_5  GTACGTGAC
#> 9       ss  tgo     tgo_ss_SANGER_5   TGCGTGAC
#> 10     tai  tgo    tgo_tai_SANGER_5  RCACGTGAC
#> 11     trh  tgo    tgo_trh_SANGER_5 RTACGTGACC
#> 
#> [Note: incomplete universalmotif_df object.]

Tidying Motif Metadata

A few reality checks

Removing duplicate motif matrices

Session Info