BettrData Customer Portal

Search

Home

Release Notes

User Docs

Customer Portal
Customer Portal
🧬

Convert Code Override

💡
o·ver·ride: interrupt the action of (an automatic device), typically in order to take manual control.
  • Overview
  • setup
  • overrideCode
  • Convert vs. Convert Type
  • Convert Type (Order: 1)
  • Convert (Order: 2)
  • Convert Type - Final (Order: 3)
  • Class Documentation
  • Holder
  • Constructor
  • Members
  • ParsedName
  • Constructor
  • Members
  • ParsedPhone
  • Constructor
  • Members
  • RestService
  • Constructor
  • Members
  • References
  • Examples
  • Change a field
  • Modify an amount field
  • Bin a field for validation
  • Copy a field to another
  • Calculate a derived value
  • Drop Records
  • Regular Expression (RegEx) Match
  • Plug a date 1 year ago
  • Plug Dates
  • Parsing JSON
  • Get year, month, day from a date
  • Email Verification 1
  • Email Verification 2
  • Filename
  • Plug Record Id or Line Number
  • split DOB components
  • Get an original source field
  • Concatenate several fields together
  • Access Client or File Attributes
  • Get CreateDate from File
  • Caps Case a Set of Fields
  • Create a Banded Range Field
  • Reduce: Keep Latest Record Only

Overview

💡
Users are able to implement more complex logic using one or more code overrides. Code can be injected at the convert level, affecting on that specific convert or at the convert type level, which would apply the logic to all pipelines of that specific type.

setup

💡
Use setup to perform an operation one time, and pass the result of that operation to overrideCode. For instance, if you pull a lookup data table from an API, rather than doing it in overrideCode which would call an API many times, make the API call one time in setup, then use the resulting data in overrideCode.

overrideCode

💡
Code run in overrideCode is executed on each row, represented by a Holder. It must return an instance of a Holder object.

Convert vs. Convert Type

💡
Convert Type code is run before the code for the specific convert. Convert type code must be enabled by checking then Enabled checkbox and saving. Other than the ordering and enabled, both operate in a similar way.

Convert Type (Order: 1)

Administration → Convert Types → Click convert hyperlink → Code

image

Convert (Order: 2)

Files → Menu for specific file → Open in Builder → Code

image

Convert Type - Final (Order: 3)

image

Class Documentation

Holder

💡

Holder is the default container used to serialize data through the conversion process. It can be changed using code overrides. Use the copy method or upsertField to mutate an instance of a holder to return.

Constructor

Members

val client: Option[Client]
val convertConfiguration: Option[ConvertConfiguration]
val errorList: Option[List[String]]
val errors: List[TransformationError]
val fieldConfiguration: Option[FieldConfiguration]
val file: Option[File]
def getField(k: String): Option[Any]
def getFieldAs[T](k: String): Option[T]
def getFieldAsDateTime(k: String): Option[DateTime]
def getMatchcode(): Option[String]
val lineNumber: Long
def removeField(k: String): Holderf
def toString(): String
def upsertField(k: String, v: Option[Any]): Holder
def validField(str: String): Boolean

ParsedName

Constructor

Members

val first: Option[String]
val firstInitial: Option[String]
val firstNameValidation: Option[NameValidation]
def fullname(): Option[String]
val gender: Option[String]
val last: Option[String]
val middle: Option[String]
val nicknames: Option[String]
val postnomial: Option[String]
val prefix: Option[String]
val soundEx: Option[SoundEx]
val suffix: Option[String]
def toString(): String

ParsedPhone

Constructor

new io.phase3.transform.model.ParsedPhone(original: Option[String] = None, formatted: Option[String] = None, isValid: Option[Boolean] = None)

Members

val formatted: Option[String]
val isValid: Option[Boolean]
val original: Option[String]

RestService

Constructor

💡
The io.phase3.transform.service.RestService class is static.

Members

def delete(uri: String, headers: Option[Map[String, String]] = None): Option[Int] //Response Code

def get[T](uri: String, headers: Option[Map[String, String]] = None (implicit arg0: Manifest[T]): Option[T]
def post[T](uri: String, body: String, headers: Option[Map[String, String]] = None (implicit arg0: Manifest[T]): Option[T]

References

Scala Cheat Sheet

Scala REPL Online

Examples

Change a field


val state = if(holder.getFieldAs[String]("state").getOrElse("XX") == "AX"){
    Some("AZ")
}
else{
    holder.getFieldAs[String]("state")
}

holder.upsertField("state",state)

Modify an amount field

val amount = holder.getFieldAs[Double]("amount").getOrElse(0.0) * 1.64

holder.upsertField("amount", Some(amount))

Bin a field for validation

val dollar = holder.getFieldAs[String]("dollar").getOrElse("0").toInt

val myBin = dollar match {
  case 0 => Some("SM")
  case x if x < 10 => Some("MD")
  case _ => Some("LG")
}

holder.upsertField("myBin",myBin)

Copy a field to another

holder.upsertField("copy_of_field1",holder.getFieldAs[String]("field1"))

Calculate a derived value

val dollars = holder.getFieldAs[Double]("dollars")
val quantity = holder.getFieldAs[Int]("quantity")

holder.upsertField("net",Option(dollars * quantity))

Drop Records

//Ensure that drop and dropReason are added as fields of this convert type

val (drop,dropReason) = if (holder.getFieldAs[String]("item_id").isEmpty) 
  (Some(true),Some("Missing item_id")) 
else 
  (None,None)

holder.upsertField("drop",drop)
      .upsertField("dropReason",dropReason)

Regular Expression (RegEx) Match

val regex = "(^[A-Za-z\\-0-9]*)".r
  
val itemId = if(holder.getFieldAs[String]("item_desc").isDefined){
  regex.findFirstMatchIn(holder.getFieldAs[String]("item_desc").get) match {
    case Some(i) => Some(i.group(1))
    case None => None
  }
}
else{
  None
}

holder.upsertField("item_id",itemId)

Plug a date 1 year ago

holder.upsertField("transactionDate",Some(new org.joda.time.DateTime().minusYears( 1 )))

Plug Dates

import org.joda.time.DateTime

val now = new DateTime()

holder.upsertField("my_date_field", Some(now))
new org.joda.time.DateTime().withYear(1900).withDayOfYear(1).withMonthOfYear(1)

Parsing JSON

Get year, month, day from a date

Email Verification 1

Email Verification 2

Filename

val fileName = holder.file.get.alias.getOrElse(holder.file.get.fileName.getOrElse(""))

Plug Record Id or Line Number

val recordId = holder.file.get._id
val lineNumber = holder.lineNumber

//make sure record_id and line_number are fields in your convert type    
holder.upsertField("record_id",Some(recordId))
      .upsertField("line_number",Some(lineNumber))

split DOB components

Get an original source field

💡
Administration → Convert Types → Output Options → Options → Passthrough Raw Input Data must be checked
# Where: field.0, field.1, field.2, field.3
holder.upsertField("test01",holder.getSourceField(0))

Concatenate several fields together

Access Client or File Attributes

Get CreateDate from File

Caps Case a Set of Fields

Create a Banded Range Field

Reduce: Keep Latest Record Only

new ReduceOverride {
  override def reduceOverride(keyRecords: (String, Iterable[Holder]),dedupeType:String): Iterable[Holder] = {
    val (key, holders) = keyRecords

    val newestFirst = holders.toList.sortBy(_.getFieldAsDateTime("date").getOrElse(new org.joda.time.DateTime()).getMillis)(Ordering.Long.reverse)

    List(newestFirst.head)
  }
}
new RuntimeOverride {

  override def setup(client: Option[Client],file: Option[File]): Option[Any] = {
      //Return a global here, run once, available as setup below
      None
  }

  override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {
    
		//Custom logic here
    val state = if(holder.getFieldAs[String]("state").getOrElse("UU") == "AX"){
      "AZ"
    }
    else{
      holder.getFieldAs[String]("state").getOrElse("UU")
    }
    
    holder.upsertField("state",Some(state))

  }
}
new io.phase3.transform.model.Holder(lineNumber: Long = 0, parsedInput: Option[Array[String]] = None, mappedInput: Option[Map[String, Option[Any]]] = None, errors: List[TransformationError] = Nil, errorList: Option[List[String]] = None, convertConfiguration: Option[ConvertConfiguration] = None, file: Option[File] = None, client: Option[Client] = None, matchcode: Option[String] = None, fieldConfiguration: Option[FieldConfiguration] = None)
def copy(lineNumber: Long = lineNumber, parsedInput: Option[Array[String]] = parsedInput, mappedInput: Option[Map[String, Option[Any]]] = mappedInput, client: Option[Client] = client, file: Option[File] = file, errors: List[TransformationError] = errors, matchcode: Option[String] = None, errorList: Option[List[String]] = errorList, convertConfiguration: Option[ConvertConfiguration] = convertConfiguration, fieldConfiguration: Option[FieldConfiguration] = fieldConfiguration): Holder
new io.phase3.transform.model.ParsedName(prefix: Option[String] = None, firstInitial: Option[String] = None, first: Option[String] = None, nicknames: Option[String] = None, middle: Option[String] = None, last: Option[String] = None, suffix: Option[String] = None, postnomial: Option[String] = None, gender: Option[String] = None, firstNameValidation: Option[NameValidation] = None, lastNameValidation: Option[NameValidation] = None, soundEx: Option[SoundEx] = None)
val customerRefStr:String = holder.getFieldAs[String]("var01").getOrElse("{}").replaceAll("\'","\"")

    val customerRefMap:Map[String,String] = try {
      io.phase3.common.service.JsonService.deser[Map[String,String]](customerRefStr)
    } 
    catch {
        case e: Throwable => e.printStackTrace() ; Map[String,String]("value" -> "0")
    }
  
    
    holder.upsertField("customer_id",customerRefMap.get("value"))
val (year,month,day) = if(holder.getFieldAs[org.joda.time.DateTime]("sale_date").isDefined){
      
      val d = holder.getFieldAs[org.joda.time.DateTime]("sale_date").get
      
      (Some(d.getYear), Some(d.getMonthOfYear), Some(d.getDayOfMonth))
    }
    else {
      (None,None,None)
    }
    
    holder.upsertField("year",year)
          .upsertField("month",month)
          .upsertField("day",day)
			import io.phase3.transform.model.EmailValidationResponse
      import io.phase3.transform.service.EmailVerificationService

      val profile = holder.getFieldAs[ProfileData]("profile")
      val (e1:Option[Boolean],e2:Option[Boolean],e3:Option[Boolean]) = if(profile.isDefined){
        
        val email01 = if(profile.get.Email_Array.getOrElse(List[String]()).isDefinedAt(0)){
          Option(profile.get.Email_Array.getOrElse(List[String]())(0))
        }
        else{
          None
        }

        val email02 = if (profile.get.Email_Array.getOrElse(List[String]()).isDefinedAt(1)) {
          Option(profile.get.Email_Array.getOrElse(List[String]())(0))
        }
        else {
          None
        }

        val email03 = if (profile.get.Email_Array.getOrElse(List[String]()).isDefinedAt(2)) {
          Option(profile.get.Email_Array.getOrElse(List[String]())(0))
        }
        else {
          None
        }
        
        (Option(EmailVerificationService.resolve(email01).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"),
          Option(EmailVerificationService.resolve(email02).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"),
          Option(EmailVerificationService.resolve(email03).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"))
      }
      else{
        (None,None,None)
      }

      holder.upsertField("email1_validated",e1)
        .upsertField("email2_validated",e2)
        .upsertField("email3_validated",e3)
		import io.phase3.transform.model.EmailValidationResponse
    import io.phase3.transform.service.EmailVerificationService

    /** **************************************************
     * Validate the client's email
     * *************************************************** */
    val clientEmailValidated = EmailVerificationService.resolve(holder.getFieldAs[String]("email")).getOrElse(EmailValidationResponse()).state.getOrElse("undeliverable") == "deliverable"


    holder.upsertField("email_validated", Some(clientEmailValidated))
new RuntimeOverride {

override def setup(client: Option[Client],file: Option[File]): Option[Any] = {
//Return a global here, run once, available as setup below
None
}

override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {

import org.joda.time.DateTime
import org.joda.time.format.DateTimeFormat

val mm = holder.getFieldAs[String]("client01")
val dd = holder.getFieldAs[String]("client02")
val yyyy = holder.getFieldAs[String]("client03")

val dob = if(mm.isDefined && dd.isDefined && yyyy.isDefined){

  try {
    val formatter = DateTimeFormat.forPattern("yyyy/MM/dd")
    val dt = formatter.parseDateTime(s"${yyyy.get}/${mm.get}/${dd.get}")
    Some(dt)
  }
  catch {
      case _: Throwable => None
  }
}
else {
  None
}

holder.upsertField("dob",dob)


}

}
    val field1 = holder.getFieldAs[String]("field1").getOrElse("")
    val field2 = holder.getFieldAs[String]("field2").getOrElse("")
    val field3 = holder.getFieldAs[String]("field3").getOrElse("")
    
    //Concatenate fields in different ways
    val concatField = field1.concat(field2).concat(field3)
    val concatField2 = s"$field1 x $field2 x $field3"

    holder.upsertField("field4",Some(concatField))
      .upsertField("field5",Some(concatField2))
new RuntimeOverride {

  override def setup(client: Option[Client],file: Option[File]): Option[Any] = {
    //Return a global here, run once, available as setup below
    Some((client,file))
  }

  override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {
  
    val client = setup.get.asInstanceOf[(Option[Client], Option[File])]._1.getOrElse(Client("None"))
    val file = setup.get.asInstanceOf[(Option[Client], Option[File])]._2.getOrElse(File("None", "None"))

    /* **************************************************
     * Setup email skip flag
     **************************************************** */
    val clientAttribute1 = client.attributes.getOrElse(Map[String, Any]()).getOrElse("client_attr_1", "").asInstanceOf[String]
    val fileAttribute1 = file.attributes.getOrElse(Map[String, Any]()).getOrElse("file_attr_1", "").asInstanceOf[String]
    val clientName = client.name
    val clientId = client.id
    val fileId = file._id

    holder.upsertField("fid", Some(fileId))
  }
}

import scala.util.{Failure, Success, Try}
import org.joda.time.format.ISODateTimeFormat

new RuntimeOverride {
  override def setup(client: Option[Client], file: Option[File]): Option[Any] = {
    Try {
      for {
        fileId <- file.map(_._id)
        clientId <- client.map(_.id)
        url = s"http://bettrdata.back:1337/datafile/$fileId"
        response <- RestService.get[Map[String, Any]](url)
        createDate <- response.get("createDate").collect { case s: String => s }
      } yield createDate
    } match {
      case Success(result) => result
      case Failure(e) =>
        println(s"Failed getting 'createDate' in 'def setup': ${e.getMessage}")
        None
    }
  }

  override def overrideCode(holder: Holder, setup:Option[Any]): Holder = {
    val createDate = setup.map(_.asInstanceOf[String]) // this is the string version of the date
    val dateTimeFormatter = ISODateTimeFormat.dateTime()
    val createDateAsDateTime = createDate.map(dateTimeFormatter.parseDateTime) // this is the JodaTime DateTime object version of the date
    println(createDate)
    
    holder
  }
}  
    def capsCaseTestMultipleFields(h: Holder, p: List[Any]): Holder = {
      // Add in a list of field names to do this on
      val fieldList: List[String] = List("person_first_name", "person_last_name", "person_whole_name")

      val anonCapsCaseFieldValueFunc = (h: Holder, p: List[Any], columnName: String) => {
        val nameOption: Option[String] = h.getFieldAs[String](columnName)
        if (nameOption.isDefined) {
          val name = h.getFieldAs[String](columnName).map(_.trim.toLowerCase.split("\\s+").map(_.capitalize).mkString(" "))
          h.upsertField(columnName, name)
        } else {
          h
        }
      }

      val todoLocal: List[((Holder, List[Any], String) => Holder, List[Any], String)] = fieldList.map(field => (anonCapsCaseFieldValueFunc, List(), field))

      def updateHolder(h: Holder, todo: List[((Holder, List[Any], String) => Holder, List[Any], String)]): Holder = {
        todoLocal.foldLeft(h: Holder) {
          case (currentHolder: Holder, (f, p, s)) => f(currentHolder, p, s)
        }
      }

      updateHolder(h, todoLocal)
    }

    import scala.util.Try
    def tryToInt( s: String ) = Try(s.toInt).toOption
    

    val age1OptInt = if(holder.getFieldAs[String]("AGE1").isDefined) tryToInt(holder.getFieldAs[String]("AGE1").get) else None

    // Example: Replace state "AX" with "AZ", otherwise keep existing state.
    val AGE1 = if (age1OptInt.isDefined) {
      val age = age1OptInt.get
      if (age >= 18 && age <= 24) Some("18-24")
      else if (age >= 25 && age <= 34) Some("25-34")
      else if (age >= 35 && age <= 54) Some("35-54")
      else if (age >= 55 && age <= 64) Some("55-64")
      else if (age >= 65) Some("65+")
      else None
    } else {
      None
    }

    holder.upsertField("AGE1", AGE1)