SFDX Upgrade Error

Recently, I upgraded the Salesforce CLI, sf, and ran into the following error:

Error (10): Metadata API request failed: Cannot read properties of null (reading ‘split’)

If you look closely at your Salesforce projects in VS Code or via your favorite OS Explorer, you will see that there are two hidden folders.

  • .sf
  • .sfdx

These folders contains metadata, tools, apex classes, and other tools. Regardless of the OS or VS Code that you are running – the error or errors that result from an cli upgrade manifes themselfes when trying to work with existing Salesforce projects.

I removed the .sf and .sfdx hidden folders from my project. Then, ran 

sf config:set target-org=myorg

I was able to work with my project without encountering any additionals errors.

Salesforce Development with CI/CD Pipelines Part 1

Overview

Salesforce supports the deployment of source code and metadata via CI/CD pipelines.  In this document, we will refer to the term “metadata deployment”, which will implicitly include code.  DevSecOps teams rely on CI/CD pipelines for integration and deployment of metadata during the development and operational lifecycle.  The key concept of “continuous integration/delivery” is to deliver value to both DevSecOps and the business continuously.  You might be wondering why I am bringing the business into this discussion?  Everything that we do in DevSecOps is based on delivering the most value upfront (Agile/Scrum principle) to the business.  If our DevSecOps teams are constantly dealing with integration and software delivery issues, then the value delivery is not streamlined nor realized in a timely manner.   This is not a good predicament to be in.   CTO’s and other leaders will have lengthy meetings and a long laundry list of actions items that will need the precious time of our DevSecOps team; thus, shifting away the focus from DevSecOps to mundane task completion and status reporting.  We don’t want to be the program that triggers such laundry list nor the oversight that comes with such predicament.

There are many benefits of having a CI/CD pipelines.  A couple that come to mind are the speed in which teams can integrate their work with other teams, identify collisions internally and externally, speed to fix defects within the development life cycle (the earlier the better) and the speed to deliver the features to business stake holders.  Yes, I am simplifying this discussion since we can write an entire article series on benefits of a CI/CD pipeline.

In this blog series, we are going to build a Salesforce CI/CD pipeline on GitLab.   Our approach will be different from the Salesforce trailheads in that we will focus on using sfdx to push our project to the Salesforce Org of choice upon a commit event in the repository.  The preferred approach is to use unlocked packages first.  This document assumes that you are not able to create unlocked packages due to the complexity of your organization (i.e. – Multiple vendors working on different applications that share objects).   

The Getting Started section will get us going with some housekeeping items that we have to get done before jumping into GitLab. NOTE: This article series requires proficient knowledge of Salesforce DX. If you are not there yet, go thru all the trailheads 🙂

What are we going to do?

This is what we are going to do in Part 1 of the series.

  • Create a Connected Application
    • Create a self-signed certificate for the connected application.
  • Create a Permission Set

In Part 2 of this series, will have the steps to configure GitLab CI/CD pipelines.

Getting Started with OpenSSL

We start by creating a self signed certificate.  This certificate is for the CI/CD Pipeline to present and retrieve an access token from Salesforce. You don’t have to get a Cert signing authority to sign your certificate request.  If you chose to go that route, the steps will be the same; however, you will have to follow your process to get the certificate signed.  Before we get started with the creation of a self-signed certificate, make sure that you have openssl installed on your favorite OS.  Openssl is available for all operating systems.  If you don’t have OpenSSL installed, take a look in the reference section on how to install Openssl on your operating system and the command to create a self-signed certificate.

Once you have the self signed certificate, you need to encrypt it because we don’t want anyone who has access to the GitLab project to use it.  In the sample project, the directory assets stores the encrypted certificate.   We will add the password to decrypt the file as a GitLab CI/CD Variable. 

Pre-Work Checklist:

  • Create a self signed certificate. Or get a signed certificate from your certificate authority.
  • Encrypt the self signed certificate and place it in your project folder (i.e. – assets/)

Encrypt the certificate

Navigate to the directory where the certificate is stored and run the following command:

:>openssl enc -aes-256-cbc -salt -pbkdf2 -pass pass:<password> -d -in assets/server.key.enc -out assets/server.key

Openssl Command Reference:

enc

we are telling openssl to encrypt

aes-256-cbc

the encryption cypher to use.

salt

is used to prevent dictionary attacks.

pbkdf2

cryptography key derivation function used to reduce password brute force attacks.

pass

password used to encrypt the file

-P

print out the salt, key and IV used

-in

input file (assets is the directory that I used in my project)

-out

output file

Create a Salesforce Connected Application

We need to create a connected application that we will use to get an access token from Salesforce.   The following 10 steps:

  1. Create your connected app, and complete its basic information.
  2. In the API (Enable OAuth Settings) area of the page, select Enable OAuth Settings.  (Refer to the image below)
  3. If you’re setting up a connected app for an external application on a device with limited input or display capabilities, such as TVs, appliances, or command-line applications, select Enable for Device Flow.
    A callback URL isn’t used in the device flow. However, when this flow is enabled, the value for the callback URL defaults to a placeholder. You can specify a callback URL if needed, such as when this same client is being used for a different flow.
  4. Enter the callback URL (endpoint) that Salesforce calls back to your application during OAuth. It’s the same as the OAuth redirect URI.  Depending on which OAuth flow you use, the URL is typically the one that a user’s browser is redirected to after successful authorization. Because this URL is used for some OAuth flows to pass an access token, the URL must use secure HTTPS or a custom URI scheme.
    If you enter multiple callback URLs, at run time Salesforce matches the callback URL value specified by the app with one of the values in Callback URL. It must match one of the values to pass validation. Separate multiple callback URLs with line breaks. The callback URL field has a limit of 2000 characters, cumulatively. If you enter several URLs and they exceed this limit, create another connected app to manage more callback URLs.
  5. Check the Use digital Signatures box.
  6. Click on the Browse button and upload the self-signed certificate. (i.e. – server.key)
  7. Select the OAuth scopes to apply to the connected app. OAuth scopes define permissions for the connected app, which are granted as tokens after the app is authorized. The OAuth token name is in parentheses.
    1. Access and manage your data (api)
    2. Perform requests on your behalf at any time (refresh_token, offline_access)
    3. Provide access to your data via the Web (web)
  8. Check the Require Secret  for Web Server Flow Checkbox.
  9. Check the Require Secret for Refresh Token Flow Checkbox.

The image below depicts the setting that were outlined in the previous 9 steps.

SFDC Connected App Settings

Next, retrieve the Consumer Key for your Connected App. From Setup -> Apps -> App Manager: Select the connected application that you created and click on the drop-down menu on the left.  Select View and Copy the key and save it in a secure place.  

You will need this key for the GitLab CI/CD Pipeline Variables (CONSUMER_KEY  & PROD_CONSUMER_KEY).  Note:  If you are deploying to multiple Salesforce orgs, then you will need to create a connected app for each org.

Connected App Consumer Key

Update the Oauth Policy for the connected app.  Failure to do so will result in a Grant JWT error.

Permission Set Creation

Next, we need to create a permission set and assigned it to the Salesforce administrators/DevSecOps team that will operate the CI/CD pipeline.   Basically, the permission set does not need administrative privileges, but you must be able to do all things that relate to deploying metadata.  The key to the permission set, is to assigned the Connected App under the “Assigned Connected Apps” section of the profile.  We will refer to the permission set as CICD Permission Set.

Then, you must assign the permission set to the DevSecOps team.  You will need an administrator or delegated administrator privileges that can assign permission sets to users.

Permission Set Menu

Testing the Permission Set & Connected App

Congratulations on reaching this section.  Its time to test the permission set and connected app configuration with the sfdx auth:jwt:grant command.  You must use the connected apps consumer key and pass it as a parameter to clientid. This command will grant the client an access token.

:>sfdx auth:jwt:grant --clientid  3MVG9GYWKbMgBvbBlahBlahBlahBlahBlahBlah --jwtkeyfile <path to>/server.key --username devops.user1@salesforce.com --instanceurl https://test.salesforce.com --json

clientid

This is the Consumer Key of the connected application.

jwtkeyfile

This is the self-signed certificate that we created a long time ago.  Make sure that you are using the decrypted version.

username

The username that was granted the CI/CD permission set.

instanceurl

The login URL of your Salesforce Org

Summary

We created a Salesforce Connected Application that we are going to use from our CI/CD pipeline to push metadata to Salesforce Sandboxes. Part 1 is foundational and you need to have this done before moving on to Part 2. There are several advance security concepts when creating a connected application. Do review the documentation on connected applications. We are creating a JWT Bearer token flow and granting the connected application access to the Salesforce API, Data and permission to perform requests on your behalf. DO NOT share your consumer key and do keep it save. Review the JWT bearer token flow, it’s important to understand how OAUTH 2.0 works and it will be on your CTA exam.

JWT Error Message

If you see the following error message, make sure that you revisit the Oauth Policies for the connect application and make  sure that it’s set to:  “Admin Approved users are pre-authorized”

Error Message:

We encountered a JSON web token error, which is likely not an issue with Salesforce CLI. Error authenticating with JWT config due to: user hasn't approved this consumer.

Reference

How to create a self signed certificate:

https://devcenter.heroku.com/articles/ssl-certificate-self

SFDX Usernames & Orgs

Salesforce DX Sandbox Config

Before you start developing with Salesforce DX, I recommend that you set your org and default username. The setdefaultusername parameter sets all Salesforce projects to use this org/scratch-org when using sfdx commands. This command is very helpful if you support multiple Salesforce projects.

sfdx cli command
sfdx force:auth:web:login --instance-url https://test.salesforce.com --setdefaultusername --set-alias my-alias
  • instance-url: provide the url for your Salesforce domain that you want to authenticate with.
  • setdefaultusername: tell sfdx to use this username/org for all projects.

How to check which username is set?

sfdx force:org:list

The output will list all Orgs that your sfdx client is connected too. The org with the (U) next to the Alias is your default username.

Tensorflow on Windows 10 – Sample Code

If you followed my previous blog posting, you should have your Windows Desktop or Laptop configured with NVIDIA CUDA and cuDNN. The last step is to test the installation and configuration of the NVIDIA CUDA and cuDNN libraries. On this post, we are going to use Tensorflow with Python to get the GPU information.

Tensorflow with Python

My preferred approach is to use Tensorflow with Python, it’s not as fast as C++ but it will validate the configuration.

The awesome folks at JetBrains have made a Community Edition of PyCharm IDE (My IDE of choice).

Here are the steps for a quick start.

  1. Install Python (use chocolatey package manager for Windows)
  2. Download PyCharm CE.
  3. Install PyCharm.
  4. Run PyCharm
  5. Create a New Python Project or Open an existing (Clone my github project)
  6. PyCharm will automatically create a virtueal env (venv), so install the libs in requirements.txt file.

You can also run the main.py file from the command line.

  • Open a terminal and navigate to the location of the project
  • Run :>pip install -r requrements.txt
  • Run :>python main.py

Output

Main.py Output

From the output of the main.py, we can see that the nvcuda.dll and the dynamic library from cuDNN are loaded successfully. If you missed a step or your $PATH is not setup correctly, the libraries will not be found.

The code ran on my laptop that has an NVIDIA GeoForce GTX 1650 TI, which you can see on the output image line #2. If I ran the code on a device with multiple GPU cards, then you will see the count to be more than 1.

The code has a sample ml model that it’s compiled, trained and evaluated. The ML Code is listed below.

Reference

Tensorflow install documentation:

https://www.tensorflow.org/install/pip

Sample Code from Tensorflow’s Beginner Docs (https://www.tensorflow.org/overview)

import tensorflow as tf
mnist = tf.keras.datasets.mnist

(x_train, y_train),(x_test, y_test) = mnist.load_data()
x_train, x_test = x_train / 255.0, x_test / 255.0

model = tf.keras.models.Sequential([
  tf.keras.layers.Flatten(input_shape=(28, 28)),
  tf.keras.layers.Dense(128, activation='relu'),
  tf.keras.layers.Dropout(0.2),
  tf.keras.layers.Dense(10, activation='softmax')
])

model.compile(optimizer='adam',
              loss='sparse_categorical_crossentropy',
              metrics=['accuracy'])

model.fit(x_train, y_train, epochs=5)
model.evaluate(x_test, y_test)

Tensorflow on Windows 10 with GPU Support – Part 3

This is the final post on getting your windows 10 desktop/laptop ready with Tensorflow (GPU Enabled). The main take away here is that after you install the NVIDIA CUDA SDK, you must install cuDNN version 8.2.2 for CUDA 11.4. Make sure that you select the version of cuDNN that supports your CUDA install.

Download cuDNN

Login or create a developer account at https://developer.nvidia.com/. Once you log in, go to NVIDIA Developer resources. The link to download cuDNN follows:

https://developer.nvidia.com/rdp/cudnn-download and click on the link for your system.

NVIDIA has done a great job with their documentation, so I am going to point you there so that you get the installation guide.

https://docs.nvidia.com/deeplearning/cudnn/install-guide/index.html#installwindows

Optional – Compile the samples and run them.

Congratulations! Your system is ready to run Tensorflow. I have one more post which has python code that validates if Tensorflow can use the GPU. Stay tunned and happy Tensorflowing…

Stateful k8s

Recently, i ran into an error with my k8 yaml file. I was defining a Stateful workload for a redis cluster and when I ran the create cluster command, I got the error below.

Results in error:
error: error validating "redis-def.yaml": error validating data: [ValidationError(StatefulSet.spec): missing required field "selector" in io.k8s.api.apps.v1.StatefulSetSpec, ValidationError(StatefulSet.spec): missing required field "serviceName" in io.k8s.api.apps.v1.StatefulSetSpec]; if you choose to ignore these errors, turn validation off with --validate=false

Looking closely at the K8 documentation, I missed the required “Selector” attribute.

kind: StatefulSet
metadata:
  name: web
spec:
  selector:
    matchLabels:
      app: redis # has to match .spec.template.metadata.labels

The snippet above was taken from the Kubernetes documentation and it shows the “Selector” attribute with the “matchLabels” and “App” attributes.

I added those attributes to my redis-def.yaml file and problem solved.

Reference

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

Tensorflow on Windows 10 with GPU support – Part 2

This is the second part of a three part series on installing and configuring Tensorflow on Windows 10 with GPU.

If you have not done so, take a look at part 1 and double check that you have the required components. Part 1

Install NVIDIA CUDA Toolkit

Now you are ready to install the CUDA toolkit. Go to the NVIDIA website and download the CUDA toolkit.

Select the your OS and download the toolkit.

Next, navigate to the download location and double click on the .exe file. You should see the installer window leave the default Temp folder, this will be deleted when the installer is done.

NVIDIA Installer

Click Ok. And you should see the progress bar window.

Extraction Progress Bar

When the CUDA setup is complete, you will get the installer wizard. Read the EULA and click on Agree and Continue.

License Agreement

Next, we select the express installation option. Express and Click Next.

Installation Option

Remember in Part 1 that we needed to install Visual Studio 2019? If you missed that step, here’s the error message that you will see.

Error Code

The next window “Preparing for Installation”.

Preparing Installation

NSight installation summary.

NSight Installation

Installation is complete….

Installation Summary

I chose not to run the examples, but if you have time I recommend running a couple. On our next post, we will install cuDNN.

Tensorflow on Windows 10 with GPU Support – Part 1

This is the first part of a three part series on installing and configuring Tensorflow with Keras on Windows 10. We will also enable GPU support.

First, we need to install the Microsoft Visual Studio Toolkit. Then, we install the NVIDIA CUDA Toolkit. Finally, we install the NVIDIA cuDNN. Also, we need python 3.x installed and your favorite python IDE (PyCharm). Let’s jump right in and get started.

Installing Microsoft Visual Studio CE

Click on the link below to download Visual studio. Then follow the instructions on installing the software.

Download Visual Studio 2019 for Windows & Mac (microsoft.com)

Navigate to the directory containing the .exe file and double click to get the installer window.

The initial installation window asks you to select which components to install.  Select the C++ option and any others that interests you.  I use Visual Studio Code for everything except Python, so my install was light weight.

The installer progress bar…

Once the installation is complete, you will get a window to login to Microsoft.

Sign in, select the theme that you like and click on the Start visual studio button.

Congratulations, you have completed the first of three posts.

Q&A

Why do I have install Visual Studio 2019? NVIDIA CUDA Toolkit requires Visual Studio. If you have another version of Visual Studio the CUDA installer should not throw an error.

SFDX Unable to Authorize Org

Featured

Overview

Recently, I ran into this error while refreshing my Oauth Token on VS Code. I tried to re-authorize the org and it failed. I had never encounter this type of failure. Usually, clearing out the .sfdx/<username>.json file and reauthorizing the org worked. This time it was different.. The error lingered and I had no additional information to debug. My next step was to research and troubleshoot.

Did I mention that I run Linux?

I recently had applied a bunch of package updates to my linux desktop and thought that my firewall was blocking tcp traffic into port 1717. I did a quick check and that was not the case. BTW – the local nodejs server runs only while the OAUTH workflow is in progress. Once the flow completes, the local server is shutdown.

http://localhost:1717/OauthRedirect?code=jedeiojropu2iou23u2io3uj

If you want to modify the port that the request should redirect, then on your sfdx-project.json add the oauthLocalPort key.

"namespace": "",
    "sfdcLoginUrl": "https://login.salesforce.com",
    "sourceApiVersion": "50.0",
    "oauthLocalPort": 7171

How to Fix this error

I logged out of all my Orgs on VS Code. Go to the command palette and look for:

SFDX: Log out of all Authorized Orgs

A quick search on the web and I stumbled on this stack exchange post, which was extremely helpful.

Stake Exchange Article

Authorize a Device

Salesforce provides many secure flows for authorizing applications and devices. In this case, authorizing my desktop (device) was the best option. First, run the following command on the terminal in VS Code. Or open a terminal and go to sfdx project directory.

sfdx force:auth:device:login -r https://test.salesforce.com

Make sure that you specify -r or –instanceurl flag. Sandboxes use the test.salesforce.com.

The outcome of a successful authenticated device follows.

â–¶ sfdx force:auth:device:login -r https://test.salesforce.com
=== Action Required!
Enter <CODE> user code in the verification URL https://test.salesforce.com/setup/connect

Login successful for testuser@test.com. You can now close the browser.

Final Step

In VS Code, command palette, select SFDX:Authorize an Org. You should see your org listed in the command box. Select the org and you are all set.

Just in case, here’s the command that VS Code runs behind the scenes. You can run the command in liue of the VS Code approach.

sfdx force:config:set defaultusername=testuser@test.com

Note: the username must match the username that you use on the step Authorize a Device.

Ubuntu 17.10 & NVIDIA Driver Support

Good news everyone!  Ubuntu 17.10 provides support for the NVIDIA Drivers out of the box.  I was not able to get the latest driver, 390.25 with CUDA Toolkit version 9.1 to work properly on my Notebook (Lenovo W541).  However, I was able to get a previous version of the  NVIDIA driver version 384.111 with Cuda 9.0 on Ubuntu 17.10 working!

Install the NVIDIA Driver

My PC –  Lenovo W541, 32 GB RAM, CORE i7 processor and NVIDIA Quadro K1100M.  The trick is to get the driver and cuda toolkit versions to work together.  This blog provides insight into what the setup constitutes.

  1. Determine the type of NVIDIA Graphics cards that your notebook/pc has.
    • execute the following command: > lspci | grep VGA.  You should see your NVIDIA Graphics card model. lspci_vga.png
  2. Start the Ubuntu “Software and Updates” application.  Then, click on the “Additional Drivers Tab”.  Give it a several seconds to load.Software_Updates.png
  3. Select the NVIDIA binary driver – version 384.11 (proprietary).  This, step will take sometime.
  4. Once the install is complete, reboot your machine.
  5. Check that the driver was installed by running the following command.
    • :>nvidia-smi nvidia-smi.png

On the top of the image you can see the version of the NVIDIA Driver, 384.111.  Also, you can see the processes that are using the NVIDIA driver.

Installing the CUDA Toolkit 9.0

CUDA 9.0 toolkit needs GCC version 6.  By default Ubuntu 17.10 comes with GCC version 7.  Not to worry, update-alternatives comes in to save the day.

Install GCC version 6 and update alternatives.

  1. sudo apt install gcc-6
  2. sudo apt install g++-6
  3. sudo update-alternatives –install /usr/bin/gcc gcc /usr/bin/gcc-6 10
  4. sudo update-alternatives –install /usr/bin/g++ g++ /usr/bin/g++-6 10

Download the CUDA toolkit from the NVIDIA site.

:>wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run

Make the download executable and run it as sudo.

:>chmod +x cuda_9.0.176_384.81_linux-run

:>sudo ./cuda_9.0.176_384.81_linux-run –override

The installer will complain about an unsupported installation.  It’s ok – follow these steps.

CUDA Installer Q&A

You are attempting to install on an unsupported configuration. Do you wish to continue?
y
Install NVIDIA Accelerated Graphics Driver for Linux-x86_64 387.xx?
n
Install the CUDA 9.0 Toolkit?
y
Enter Toolkit Location
[default location]
Do you want to install a symbolic link at /usr/local/cuda?
y
Install the CUDA 9.0 Samples?
y
Enter CUDA Samples Location
[default location]

Check your installation by running the following command:>nvcc -V

nvcc.png

If you see the CUDA compiler driver, you did it!

Congratulations! You have installed the CUDA Toolkit.  Now let’s move on to the CUDA Configuration section of this blog.

CUDA Configuration

The CUDA toolkit provides plenty of samples to build and test the NVIDIA GPU.  Now that you install the driver, make sure to set the PATH with the cuda binaries.

open your ~/.bashrc file and add the following line at the end of the file:

export PATH=$PATH:/usr/local/cuda-9.0/bin

Next, you need to add the CUDA libraries to your environment.  This step saves you time when compiling CUDA samples or code. You don’t have to set the LD_LIBRARY_PATH.

  1. sudo vi /etc/ld.so.conf.d/cuda.conf
  2. add the following to the top of the file:/usr/local/cuda-9.0/lib64
  3. save the file and quit vi
  4. sudo ldconfig

In theory, you are ready to run the CUDA samples.  However, I found that in order to run the Simulations, that you need to install the following dependencies.

sudo apt install libeigen3-dev libopencv-dev libceres-dev libcgal-dev libboost-dev freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa libglu1-mesa-dev libglfw3-dev libgles2-mesa-dev

Now you are ready to run the samples!

Running CUDA Samples

Before running the samples, you need to pass the Gencode arguments to the compiler.  Here’s a great blog to help you figure out which Gencode arguments to pass to the compiler.

Navigate to the ~/NVIDIA_CUDA-9.0_Samples directory.  There are 8 directories that contain NVIDIA CUDA Samples (0-7).

Go into the following directory:> cd ./CUDALibraries/cuSolverDn_LinearSolver 

Open the Makefile.

Search for “SM”.  You should see the following.

gencode.png

You don’t need all the codes, just the one that applies to your NVIDIA Graphics card.  In my case, I don’t see the Quadro K series, so I used gencodes 50 & 52.

Once you are done with the Gencode Arguments, save the file and exit the editor.

From the command line:

  1. make TARGET_ARCH=x86_64
  2. run the program: $ ./cuSolverDn_LinearSolver

final.png

Try running the smoke particles example in the directory 5_Simulations.